DFFML Weekly Sync, 24 Mar 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Weekly Sync 2020-03-24

Description

Meeting Minutes: https://docs.google.com/document/d/16u9Tev3O0CcUDe2nfikHmrO3Xnd4ASJ45myFgQLpvzM/edit#heading=h.qkoqwtl92zpo

A

B

Thought this was not a single command, it was.

A

Oh yeah, understanding.

B

On my part, yeah.

A

So no worries, no is that's an easy thing that confuse I.

A

Take this opportunity to go and start start updating the median minutes here due to 24th.

A

I'm due an amnesty handwritten, digit digit demo.

A

B

So this config load, like you, told me to load one of the files from the dogs right.

A

Yes, so that was the other I think that's the only other like sort of open thing on this. It's basically yeah just grab that file path right, so you need you, you want to I, don't know if that's exactly the right, that's exactly the.

B

Right it gives I did I record gives me another like.

B

A

Yeah man will want to get rid of that code. Block console there.

B

Okay, so here it won some bytes iodine, oh so spike, oh yeah.

A

Oh, so, do sorry dot read bytes then it'll basically be.

A

Read but it's here, I updated the comments. If you refresh that.

C

A

There we go so that all looks good.

B

That thing that was the these very.

A

B

Changes right now in five or ten minutes, sweet.

A

Okay and so then reload this page, real, quick.

A

And then you should see a dot read bites: okay, so yeah. So basically that will give you the bytes from that file and then you should be good to pass that into load B, so just get rid of their weight. Dumpy replace it with this whole mess here and as long as that path is correct, I think it's correct, because the end of this should be test so test dot would be PNG PNG.

A

Would we config loader config letter dot would be the root of D F of now and then examples, amnesty, amnesty much one dot replay, so that should give you the right image there and basically, if it works, it gave you the right image if it didn't or well well, you should do here's you should you should print out that block and then like paste, that block into the file here and then do an assert equals ons. You know don't global standard array.

A

There there's some global array there that that contains the yeah I'll put it whatever the bytes were, and that way we just make sure that it's always right.

D

A

No problem, thank you. This.

D

Is looking great I think we're almost done.

A

Right here like this, should be it and then we'll have this demo done. Yeah.

B

Five to ten minutes and then.

A

Good sweet: alright, hey thanks section!

A

So, let's see what else we have on the docket here. Let me go back and Augen I can't remember what we were sort of talking back and forth. I think I lost track of something that we were talking about. I also haven't gotten a chance to read everybody's private messages to me. Yet it's I've been swamped with things, but I will be doing that. First.

D

A

D

A

You want to know when you should cancel them.

D

A

So also we I can it's kind of hard to hear you so there's some kind of mic fixing we can do there, but, okay, so I know yes, that's much better.

A

So wouldn't it cancel those so I think that that that becomes more of an issue actually with those async iterator operations like where they may just live forever, but these ones like you know so basically we should have. If you control see the thing it's going to stop right and it will run all those cleanup.

E

A

And it may know and run the output operations, but we should probably have this. Isn't this I couldn't quite figure out what what to respond to this, because it's sort of like a larger thing and it's kind of a separate thing, because we could have run into this situation even without these operations without inputs right, you could have some operation that, like it just gets, hung right and and so what this it's sort of like a separate issue. We maybe need to make here so, let's so, let's make a record of this.

A

So this is what so, when slash, how do we, okay, where I'm sharing great so it's eight canceling of running operations, slash data flows, because that's really what we're thinking right here in sort of a grand scope right. So what happens if we have?

A

If we have this data flow, that's running and there's some operations that haven't completed yet so, therefore the data flow hasn't completed, and then what do we do with it? If all of a sudden, we want to stop it like a graceful stop, and so we need to basically I think this is something that maybe we need to add to the config of the orchestrator or something or maybe it needs to end up in the config of the organist reddit context.

A

So because the orchestrator context, I believe, takes it data flow and it's gonna fig. Let's just verify that, let's see it does.

A

Yeah, okay, so remember your extruder context: config takes a data flow; no, it also takes reviews. That's right!

A

So, okay, let's see yeah, so we probably want to add a perimeter tool, a config there and then, let's see so we probably want this to be configure.

A

Okay, hmm so 2 theta fill itself, and this is kind of so what I'm thinking about right now is like it's kind of like the same thing we ran into with the validation stuff. It's like, okay. What scope does this stuff act at right, because we don't want to put it in the wrong scope and then we end up with something that doesn't like you know, that's that doesn't transfer around well as we change things, so the data flow is responsible for defining like what happens between the operations. The orchestrator is responsible for.

D

I abandoned it with some sort of code or something. What are we planning here for.

A

D

Four locations like what are we I keep any. What are we aiming for.

A

What, in what context you mean.

D

Like this pony's, like yeah, we wanna have some Stallman today so like it was yeah.

A

What's our criteria for stopping yeah, so it I mean you could basically have like so there's there's a few things right well, but there's a time out thing right and at some point that will need to be involved right and that might be something that gets specified. Unlike the there. My.

D

A

Like in in the operation definition itself with an no dataflow yeah yeah, that's probably true yeah, because then you might. This is something you would likely define when you write the operation. You would say this is the time out for this thing and then there's other things like so then, then you would have other things. I think like I think this might need to go at they. So let's just make something else here so time out time out could should be configured in.

A

A

A

We're planning to have what, like.

D

Note says, like we have in the distributor orchestrate the thing will have no serving operations right, yeah.

A

Sorry, I'm missing it's it's as I say no, it's serving. What are you saying like.

D

In the distributor orchestrated thing like, we will have a No, which is our wheezing someone with three set of operations: yeah, that's what you've been played, so you put in the notes also need to stop up to something because, oh.

A

That will be a separate thing that would be taken care because we're looking at so in this context we're looking at like what is the data flow running? The notes themselves will be like managed by the orchestrator in that in that situation in this, in this situation, we're just talking about like for this data flow when these operations are running like when should I.

A

Consider you know like when is is their yeah like you know, if this, if this operation stops running or like, if this operation stops producing outputs and it's one of those that if it's something that just stays there and produces outputs like and keeps doing that like the?

A

If we're looking at the chat example like okay, it's a chaplain, if I with that thing, it's there's, probably no timeout on that one right and but you know say there was right, then F after some time we don't see any new chat messages, then we stopped the whole datum or then we killed the operation right and we probably need several different modes. We need like the well.

A

Let's see we need this is so this is there's gonna, be a lot of exploration that happens here right because it's gonna be like what what makes sense and so their needs.

A

Well, one thing is: we need to have like a timeout on an operation right and that's sort of a separate thing, so it's sort of like a separate thing from this in itself, right, because this is for an operation so for operations operations we need a timeout because, like this could be something where this could be something where, for example, we have an HTTP request that gets made, but it doesn't complete complete within.

A

Time out period.

A

Then we need to have, we need to have something.

A

Which tells us do we rerun the operation like do we? Do we keep trying if it doesn't happen within the time out or do we raise some sort of like? Do we raise an exception or if it doesn't happen within a few times, do we raise an exception like we're going to need extra properties on this as well?

A

So this is maybe something that we want to look at, or maybe make an issue for and and look at sort of later, because this is going to come into play here, but it's a bit different than I think what we're talking about with the whoa, with what you're going to hit with the async adorable operate operations, because so so the reason why this is like something that I.

A

Obviously this is it's sort of a longer conversation, and this is why I couldn't tie it back to you quite yeah, so because what we have here is we have this situation, where we have properties like a time out on an operation where we might want to.

A

We want to be able to specify per operation like what do we do if X happens like how did its the dataflow treat that for this operation, and then we have other things like okay, well, I've controls like I, can I'm running the dataflow via the command line and I hit control-c right. So let me just write this down. So if running data flow from command line and we hit control C we're gonna have currently we run all the cleanup operations and then we just call it quits and then or what does it happen?

A

Yeah anything. That's already returned through right. If you have multiple contacts running so any contacts that are in progress, we run the cleanup operations for. But if your context is already done and it's running put in the cleanup stage, then it's done right.

A

So all can't we run all cleanup operations for all input set contacts which are haven't entered the or which are still still in the processing stage.

A

So now what we're thinking about here is like okay. Well, what if we in this case it's like? Okay, we hit the ctrl C and we probably want to go straight to the output stage in this case right.

A

So in new case, where we have a long-running operation which is feeding inputs to the data flow at any time say from the chat we want to want to be able to say if these so for, let's see, we want to be able to say like if control C is hit, okay data flow is told stopped, then these operations or so, and if this set of operations set, would include B the chat, the wait for for chat message, operation, operation.

A

Only consist of in this case, so it would only so this in this case right. We have some sort of property on the data float level, probably within the data flow itself, because this is like a large part of defining like what this data flow does. Is we're saying?

A

Okay, if at any time we're told to stop the data flow, and this thing is still running just go right to the output stage, don't go to the cleanup stage, because this thing will run forever, so basically we're assuming that these kind of operations will be will be running forever and, let's see that actually may be a better thing to put in there yeah. That's like it's hard to figure out. That may be actually a better thing to put in that data flow or the into the operation itself right.

A

So if we look at these timeout things, so the time at is at an operation level. This is probably also at an operation level right now, because when we write this Gator operation to pull from git er, it's I mean it knows that it's just going to keep pulling from getter until gator goes down right so or you lose connection right like it's always just gonna sit there, so maybe maybe okay, let's just put this here- we let's see we want to be able to save ctrl C.

A

Then this set of operations needs to or is okay, that they haven't completed, don't skip the output stage.

A

Okay, it's like don't don't go to the cleanup stage before.

A

Before or without running the output stage, okay, so we'll define this property so like whatever this property is called, maybe like the long-running or like I. Don't know like like oh there's, something that that GCC uses so GCC does not return. What is this called? Maybe we can borrow this.

A

Borrow this syntax here and that way it makes sense, no return.

A

Okay, yeah! Maybe we should call it no return and this property no returned, but then the operation.

A

Alright, so that I mean that's I, guess that's what we would do here and so then, within the data flow, we would go through and look at all the operations that we're loading and have this no return property, and then we we treat it like that.

A

We treat it like if you get the kill signal and still run the output operations. If this is one of the running active instances right.

A

So if we're told to stop at any point, then if the only the only operations that are running are our operations with no return return set, then let's see that are running our persons, never try and set. Then don't then then just stop them and go to the output stage rather than behavior of skipping to clean up stage.

A

Alright, okay, I think does that make sense? Does that sound, reasonable, okay, cool yeah I think so it seems like this is yeah. This is kind of it was I was having a hard time figuring out. What to say about this. So I think that we arrived on something that that would work here.

D

A

We want to give it some more thought right before go implementing it, but let's so, let's make some issues for this, so yeah and I don't know exactly how we're going to describe this in an issue format, but so, let's make an issue to to track.

A

What should we call this? Maybe like this like operation properties or something like operation.

D

I'm, oh yeah, you might be yeah.

A

They're they're both they're, both separate issues. So let's make issues she's got issues Wow, okay, to track this and I think think these attributes should go in some substructure or should they maybe they should? Maybe they shouldn't I, don't know. So the thing is what I'm thinking about right now is the fact that, let's see them examples should I should I should I deploy.

A

Okay, so and we're looking at these operations right so far, we've got inputs, name, output stage and that's sort of like it's pretty manageable.

A

We could start adding other things at this top level here or we could make something: that's like extra and then puttin stuff there, but maybe we'll just we'll just keep them all at the top level for now and we can move them into some substructure later if they get out of hand. So let's make issues to track this. Just call that good, okay, okay and then was there anything what? How else are things going for you again.

D

A

Yeah, let's pray, let's pray hold off on that thing and then we can.

D

The tensorflow estimated English, oh.

E

A

The error threshold on that on there tensorflow opinion.

E

A

D

I noticed that.

A

Too yeah I think we still need to.

D

A

Yeah I mean okay, so that the usually we just all, we really need to do on that. It's just rerun the that's very annoying I know we need to just rerun the rerun. The CI. If you see that happen, I mean I, always check and I notice that if it says model tensorflow, it's usually what it is, but yeah.

A

The only reason why I I hesitate to take out that test or lower it further was because that one time when we, when we did do that, transition from tensorflow one tensorflow two, we noticed that that things were messed up because of that test, so yeah well annoying. It's gonna stay around the land of this of it. But, let's see so I think this is good to go. Then yeah.

A

Let's anything else on this poor quest, specifically oh yeah, this one was sort of separate, but and then did you say you must you just want to talk about distributed stuff so.

D

A

Yeah, let's hold up on that and and hit some other people's things, then so cool. So who else is on that wanted to talk about stuff, I assume everyone, but who wants to go next?.

F

Hello, John, okay,.

A

F

A

Yes has gone, yeah.

F

People beauty than a double cast: oh.

A

F

A

I was just gonna look at this again. I saw your recent comment here and you said I guess: I don't quite understand what was going on then. So.

F

Now it's loaded into the test, I didn't get. You know, file what we were storing in the Downloads and you know. Oh.

A

You mean like, why would we change these paths to be okay, so yeah? So what I was thinking here is what we'll do is we'll change this format here to be from tests, slash, cargo audit, download to test download and then cargo on it, and that way we put all of these things under test, slash download and we could just list test. Slash download in this kit ignore file and now will also let us do so. There was this thing: I did recently here, so we added so you can use it.

A

There's a cache functionality within the github actions, and so we can do is we can take. We can take this okay. Well. This is I'm, not a very helpful example here, but we can. We can specify a path, and so we can specify slash examples. Slash should I slash test, slash download and then it will just grab everything under there and that way, since we have this cache download function, it will pull from the cache and it should speed up our CI test by a lot. That's why we wanted to do that.

A

So that's it's sort of something that that we could do I think did I say and it's separate PR yeah. So this is just sort of like for next time. I wanted you to know like once we finish this PR, then we can start doing that in the next ones.

F

And I was I think that I shouldn't should I do all the for the previous work that I have made to also test on yeah.

A

F

Put them all in there, yes, so.

A

After we put so basically we'll we'll merge this one here, so let me make some notes here so for pyaare. True for your weight, 241 no I can't be right now yeah. This is for issue two for 468, so PR 468, which is should I rest or cargo out it support.

A

We, let's see or well. This isn't really for that PR. So so, for for examples, should I download caching.

A

In future PR we want to move all of the downloader, the cached downloads oats into the examples. So I should I slashed s, slash download folder. That way we can't speed up speed up the CI by having them yes and then for PR this okay. So are we clear on that done? That's just a future PR!

A

So we'll get this one merged and then we'll do that, so that should be sort of your next course of action there and then on this one. You were saying so. What were you saying on this one? Did you try.

F

I just came to it from the PR; they come in kid. Okay,.

A

So so you still need you still need to make those okay.

F

A

And then yeah I think you saw what I said here. Basically I think this I.

E

A

Got copy pasted funny, yeah yeah, it looked good when I download it and I extracted. This looks like the correct file path, so, like I think it will work but yeah that just obviously so this stuff, this stuff is your. It says cargo cargo a bit on it, but you're downloading rest, so you'll just need to add the other one Add to download to download hirato itself but yeah. This looks good sweet, nice job.

F

Also I actually added a curated list called the language. I am eating. Oh.

E

Great great great.

F

You mentioned it as well, will not be a good option for Eddie. Oh.

A

Yes, oh well, I think the thing is about SQL is: is you know, you're not really gonna run into SQL like in a you know, flat file very much right unless it's some sort of dump of a database and those are usually automatic dumps and so I, don't think you're! It's it's gonna. Have you it's? It's really just not going to get used that much right. So we want to target things that that are gonna, get used a lot right.

A

So I think things like you know, main main. More mainstream languages, like oh I, think PHP might be a good one to there should be I. Think there's this thing that framework that people use called.

E

Composers and stuff.

A

It should be someone some tools around that, but yeah was there anything else on that front. Oh.

F

A

F

A

See almost done just need to add I'll just need to.

A

Audit download and modify rest all right. So, let's see what else do we have here? Go back to work you here.

A

So okay, let's see and I what was your skip on github? What I'm sorry I, don't know how to say your name. This is your first first call with us right.

C

A

Do you pronounce her name.

C

Sudan Sudan.

A

C

A

Right great, hey nice, to meet you Sudan shoe, thank you for for jumping on the call with us. So let's see, let's check this out here, yeah, okay, so this was and I'm sure we've all run into this.

A

Basically, these files write these generated files. I was thought it was important to check them into the git repo at first and now, I'm realizing it's just resulting in us having to run a Doc's script and and then commit the changes. Well, in fact, the docs script runs the generator for these files. So it's kind of pointless, so we just we don't really need them. If we change the, if we change the doc strings, they will get regenerated and rebuilt anyways. So I think we are all good here to just remove these.

A

This looks good. Let's just capitalize this, let's see I, guess but just to say, stay consistent, yeah, let's see yeah I, think that's perfect and then you've been working on various other things. Did you have anything they? That is like your next next steps or anything that you I.

C

Wanted to ask about the ji-suk project like yeah same proposal from the dashboard? Oh yes,.

A

That's right so I've been I've been having some issues me and the other many other mentors we've been having some issues with the dashboard I'm gonna try to get that cleared up. That is the next thing. On my on.

E

A

Docket to do is figure out. What's what's going on with all that, so because there's there's there's two things going on: basically there's a Python, GSoC dashboard and then there's the Google dashboard and we're having some issues getting everybody on both dashboards because you have to be on both dashboards this year. So it's a slight change from last year, but we're going to we're going to get that that all figured out and I just got an email from Google that the you period has been extended by another week.

A

So your submission period is still through the 31st for students, but we will be reviewing things through another week after that I believe. So, let's check out the timeline here and let's see, let's the application review could I, don't know if they've changed this yet, but I got an email. I will I will make sure they will make sure you guys know as well, if you're all locked in and signed up and stuff but yeah. Let's.

A

Tours for GSS CGS see I, know you guys, so you guys have been submitting your proposals in your draft proposals and if you have feedback that you want I believe you can submit a new problem, so you can't leave it at draft stage like you have to submit a final proposal. If you want feedback on your proposal, you should say, like you can say to me, you can say to there's other and there's Sudarsana and Rahul and Arvind is also going to be a mentor and Terry is like a half a mentor she's, sometimes serious.

A

Sometimes no, but she's got a lot of things to do so, but but mainly like, if you, if you say to me, I, will broadcast it to the other mentors. But if you say to them they will broadcast to us, but so it doesn't matter where you say it. If you say please review my proposal to one of us, we'll make sure it gets reviewed and if you've submitted it through the site as soon as I figure it out today, it's it's will be on the same page and we'll look at the draft once so.

A

If you have any specific things, you can message me and I'm making sure all them other mentors are getting the same message so that we can all provide feedback and then I'm. So what I'm doing is I'm combining the I'm. If a student sends me a proposal and says, can I have feedback, I'm broadcasting it to the other mentors? We are all giving our feedback and I'm sending you back the feedback that everybody did and that way it's not like. Okay, we're we're all sending each other different things and no one's on the same page.

A

It's like okay, if, if somebody says something all the mentors do feedback and then I get back to you, which means that it may take a little bit longer, but we will give you'll get everybody's feedback and so just make sure like that you do submit the final.

A

Like because you have to submit a final proposal, because we can't accept proposals that are still in draft status, but obviously you can't you want to wait to submit the final proposal because you can't change it once you've submitted it, so you want to leave it as a draft up until whenever you're comfor comfortable leaving it and then submit it right and so I'm gonna try to I'm going to try to loop back on everybody within the next couple days and make sure that we reviewed everyone's drafts that that has submitted them up until this point, and we will keep doing that if you, if you want review on your draft so yeah.

A

Basically that will be very soon here. It's my next action item after I finished I think I've almost caught up on everybody's full request here, so yeah so GSoC. Let me say some have o be reviewing starting review of everyone's drafts.

A

A

A

Weíll mentors will keep you posted.

A

Yes, all right, so yeah no worries there, I'll, let you guys know so I will. Let me just just so that I'm held extra accountable here. John will let everyone know when we've got all the mentors mentors.

A

Correctly configured and all of the portals, okay, so all right, let me make that an action item for myself.

A

Sign to me all right, great, so, okay, anything else, GSoC related, since we're on that.

A

Maybe random GSoC questions or anything I think you guys have all so far. You've got you've got the hang of things. I mean this is this is the way it works. We work on stuff, we submit proposals and then we continue to work on stuff, so yeah.

A

Okay, all right is that, did you have anything else you wanted to you? You just review the draft or.

A

Is there anything else you wanted to to talk about yeah.

C

So I also wanted to talk about the plate ml of wrapping. Oh.

A

C

So, like I have a like ideas for like wrapping plate ml in the g-shock period right now, I'm feeling that the wrapping the autoscaler on will be like a great fit yes or should I add wrapping plate ml as stretch.

A

C

A

C

A

That's that's very appropriate yeah. The other thing about wrapping wrapping Otto SK learn is that we have a lot of this stuff for Sai kit and it looks like they copy a lot of the psyche.

A

So have you been looking in model Sai kit and we may we gonna find this when we look at your draft and stuff but you've been looking in there, we do a lot of like automating the wrapping of the circuit models, and so it looks like they have consistency on their docstrings and class definitions and stuff that we can reuse the auto wrapping. We may be able to reuse the auto wrapping so we'll see, I mean we'll see how that goes right, but yeah and then basically so.

A

Basically, my point with that is, of course, if it turns out that the auto wrapping is pretty consistent there, then it's pretty much going to be a case of like copy paste, psych it and then move on to plat ml. Obviously it never works out quite that easily. But you know if there's there's, if that is the case, that it's mostly compatible, then that's, it's probably gonna. Be most of it, so yes, I, think that's a great great idea to put that as a stretch goal there.

A

C

A

Thank you, okay and then so, who else do we have here? Let's see who else wants? Who wants to go next.

B

Hey, hey John, Himanshu, hey.

A

I'm not sure how's it going so I.

B

A

This looks pretty much what.

B

A

Yeah, let's see I, think, oh, let's see! Oh no did I, not! Oh damn I forgot to submit this I'm. Sorry, okay, so the one question I had here was basically, why did you? Why did you capitalized sentence ID in words.

B

Yeah, because the thing is like our basic format is we'll have one features and there will be one predict right, but in this case we can have features, but only two features are required. All this one is sentence ID and the other one is words so.

E

B

The if we don't give the user like specifically that we want sentence at the end words, then anybody can enter anything. Basically, let.

E

B

Put model features then any features they can put right. You.

A

B

Was thinking maybe just sentence at in the woods? Okay.

A

And now we're yeah, because because, or else they would just they would probably.

B

A

Yeah we don't we don't want to. We want to make it as point-and-shoot as possible. That's great Wow, but I guess just why're. They capitalized then I.

B

E

B

Week and I just thought: okay.

A

B

A

Make yeah, let's just make them lower cases in case, because these are court right now, they're the only things that are that end up being capitalized and then the other thing would be that that that sentence. Id is okay. So this is a problem with the unified config stuff right and you've noticed, luckily that you have to do underscores but yeah, that's a that's pain! I'm! It's on my list! It's on my list, but it's it's a long project.

A

So the thing is that that yeah, any things with underscores well they're good for, like these very extended things that that people may not use that often I. Think since the sentence ID and the words are probably they're going to be used every time with this right and it might be better to go with like s ID or something, and then in the in the help text.

A

Explain that it's the sentence ID, because just just because that way you know if you're typing things sentence, underscore ideas, it's not as convenient to type as as s ID, so yeah, so unique, ID, yeah. And so then, if you just said it would just show up as s ID here and then words here and I, think that would sort of that would make it it. Just is a little a little bit more user.

B

Friendly man, you know.

A

Yeah other than that, this looks perfect. You've done an awesome job on this. This is like this is Wow great job, yeah yeah. This is uh there's a there's, a lot going on here, yeah yeah, that was this is cool stuff and just for my own education, I think everybody else on this call. So you have I'm not super familiar with all of these.

A

This kind of an LP models- that's going on here, but so you declare you provide it with these training examples, and then you, you have some test examples in it and in this what you did was, let's see if I could pull down to the.

A

Where are the examples here? Where did they go? Where did they go? Okay, yeah so where's the data? Okay. So this is the test data right, so Oh where's the train data.

A

Okay, so in in in the training data, what you've got for us is, we say the sentence ID, we say the words and we say the tag and then what comes through here is so you're. Basically tagging every word and you're saying these words belong to these sentences right.

B

Yes, yes, this is because we will have to work on sorts, because we can directly read the text, but that will we can't do for now. So for that what we need to do. This is a specific format when it comes to named entity recognition. So what I have here is I have a sentence. Dfl models are cool. Okay, and this is a single sentence sentence number one. So that's why the sentence. Id is one one one, one, no.

E

Yeah then, when.

B

It comes to DF FML. This is a miscellaneous word, but B means the beginning of a miscellaneous word. The.

E

B

Is inside of a Miss in his word, R is like o is nothing not.

E

B

E

B

Similarly, I have just put in something okay.

E

B

The second sentence, then, a FFL models can do classification, so this is sentence.

B

A

With their first okay, so the ne and mu R is its that's. The specific type of I guess like weird like this is something that and obviously I'm not familiar with the NER stuff. So this is like something where, if you wanna, if you, if you, if you wanted to figure out what all those tags are, you would go to this.

A

You know you would go to hugging face transformers and they have all the tags for you and, and they would tell you sort of more in depth what what how we should be tagging things in sentences and stuff.

A

Right or I'm, just thinking like I've got that I've got I I think it you've shown the idea here behind like what do you? What do you do? You know here's, here's, how you provide the sample, training data and then the test data, and my assumption would be right now, with the training data and the test data you've got the same. Examples in the training are the same ones in the tests right.

A

So if you provided different different sentences and the tests it would sort of, it would do its best right, it would do based on the training model. It would do its best to classify which ones are I, miss core being be miss core, oh right, and is there any like? Is there where? Where would people find like sort of all the all the tags that they could possibly use or maybe like something something like are there? Is you doesn't ride? Hey sorry, I think you might have cut out there for a second.

A

So so is there a? Can you hear me how about you.

B

B

A

I think yeah, so I was just saying it looks like so you do. You have pre trained in here as well, right or oh.

A

Well, alright, well, I think he's having network issues, but I think I think he does. I am just wanted to make sure yeah all right. We don't know here, MA, true hello, yeah.

B

A

Okay, I was just thinking. I was just trying to ask whether maybe you had some pre trained models in here, because I thought I'd seen that in sure enough here it is so you do have the pre trained models as well and- and it looks like.

E

B

A

Cutting out you're cutting out really bad.

A

You're you we've been cutting out really bad, so I can't really hear anything you're saying but I think.

E

A

I think you're back now. Actually so did you want to.

B

Say what you were saying yeah so I was saying these. These models are already building in their transformers, yeah.

E

B

Just have to like the dictionary, and they have built everything so I just have to pull from there. Okay, just protect it. Oh great.

A

Great, so, basically, if people want to know I guess what I'm getting at is is, is it clear when they look at so that we say it's implemented using hugging face transformer tensorflow based models? Do we want to, and we've got some little examples here? Do we want to maybe link to where we can show them? Okay, like here's, some free trained ones right, give them a little more information and say something like you know. These are implemented.

E

A

Phase transformers and here's the link where you can find all the things that that hugging phase transformers can do and here's you know, for example, maybe provide another little example that you wouldn't need a test for I. Just say something about: okay and here's, a pre trained model or something right, and here's an example of how to how to pick from their little things. Alright cool, so I'll just make a little note on that and.

B

F

B

F

A

Go on this yeah, so, let's also link to where users can find more info on existing models and other usage info from main hugging face. That's lawyers.

A

Doc's I can't spell today all right, great and then I think that is good to go all right. Thank you. Is there anything you need sort of on deck stuff.

B

Yeah they order whatever whatever thing, because that is stuck. Oh.

E

B

E

B

I'm easy, like anyone, you get yeah.

A

That's that's a it's a con, so what I'm gonna try to do with that is basically just get Conda in the main environment so because we're obviously going to run into this again. So that's that's my planner. So just one more example: let's see all right, let's see just one more example, pre-trained and and then references to talks, okay and then so what will open rabbit? Well, that John needs to figure out and CI all right. Yeah I, just I'll put this here again and I know.

A

I said I'll get to this before, but I will try to get to this I've just been I've been I've, been swamped. I have a I have a lot of things going on, but yes, I will I will I will try to see this see this one through here so because that would be, then we could get that close and done no one's ready to go so yeah. Alright, great, let's see so yeah anybody else. Who else we have Hashim I. Think.

G

Yeah, hello, hey.

A

G

Great, how much you good.

A

I can't remember where, where you were up to when we left.

G

A

G

We want to do dope test or absolute base plusses. Oh yes,.

A

That's right so.

G

I, there are a couple of ways you could force your at this class to instantiate. If you, you know directly wanted to run tests on it, yeah.

E

G

Make that much sense, because our absolute base prices won't be doing much on their own. No.

A

G

So I had tried, you know getting their stuff classes, but there are a lot of dependency. You know. Oh yeah.

E

G

You are taking their context class. You would have to take the original class as well. There are a lot of dependencies. You will have to figure out to get that face. Class yeah.

A

G

I think you worry about that. I.

A

Think what maybe I've been thinking about this a little bit and I think that so in some of the tests we have like fake model and things like that and then we'll, basically just have them return dummy stuff, and so what we could. What we could do is, let me see if I can show here. So let me just make a note here. So we're talking about doc, two simple examples:.

A

Abstract in act base classes.

A

How do we show examples for them so fake? Oh, yes, okay! Let me pull this up here.

E

A

Can I get you actually to mute if you're not talking here, because I think I mean I, think we're getting some feedback and I know my my cat's feedback, so we're gonna matter which we're gonna end up with double feedback yeah. So sorry for everybody on the recording, I know that I'm hoping it gets a little better this week tried to mess with my mic but anyways.

A

So so we've got this fake model and fake model context here right and so what we could do is we could define these guys and so say, for example, I'll just sort of show you what I'm thinking here we take these guys we move them into. So, let's see if we look at docks, you'll notice that dock test header includes all of the things that we might, we might want so ooh. Actually, oh I'll! Might this go mm-hmm?

A

Let's see yeah this'll work so basically doc. Test header is something that if you look at docks, docks just gonna fig know coughed up he. Why so look at docks comped up he? Why so the way that this works is Sphynx loads, this file, and it takes this blob of text- that's actually Python, but in a string, and it uses that as the first thing in any of the doc tests like so basically what it'll do is every time there's a doc test.

A

It takes that that those strings out of the doc tests like then the interpreter stuff and it and it it prepends. This header I, think I think it makes a new file and then prepense this header, or maybe it just like grabs that header and paste it all in there like loads it into the interpreter first or something I'm, not sure exactly how it works. But basically, the end result is that anything you put in this doc tests global setup variable as it as a string.

A

So usually it ends up being something like this like in, like from DFO from el import star, is that most of their documentation shows it like doc, test talk--I test global setup equals. So this is how most of the Sphinx documentation shows this.

A

So, to make this a regular Python file, we basically just said okay load, read: read the text of this doc test header file and call that good, so we could do is we could put something in Doc's and now at this point, we'll probably want like a subfolder for all this crap, because we're gonna do something else. Just like doc, test header and we're gonna do something like you know: fake model, dot, py and we'll just put this fake model stuff in here and now what you could do is at the top of you.

A

Could you could call this like? Instead of fake mall model, you could call it example model right and then you could do you could take this stuff and you could like you could include this file with a literal include right, so you could go into like.

A

Let's see what is that Doc's M Doc's API model base right and you could go here and you could say literal include- include and then this file right, so slash, dot, dot, slash, examples, slash or not examples, Doc's, slash, sick was fake model, dot, py right and so then we'll say something like ok, our example. For these the following test is going to be blank, and then you know we show them what's the example and then within the doc tests. For this thing we're actually going to use this example here and you'll.

A

You know you'll call example, model context and stuff, and so that way they see. Ok. What is this thing doing? It's this it's dumb right, but it gives us enough. It gives us enough of an idea what's going on for these base classes so that we've got the reference code and then, within the abstract base codes example, Docs were actually or like within the abstract base classes.

A

Docstrings were referencing the example model stuff and then so what we do here is we basically just say you know at the end of this we'd, say doc, test global setup plus equals you know a new line, new line plus read fake model right and now we just keep doing this, for you know fake source and stuff, and that way we've got we can. We can give a little demo with something that doesn't do anything show them that here's, what we're actually using and then within the the the doc tests for each method.

A

We reference the example model, or example source, and they the user, then sees ok, here's what it would look like for any given model or source calling these methods- and you know with these dummy methods- does that sound? Does that? Does that seem seem like a good plan for you from what you've seen as well or does that seem like it may not work, or this was just sort of what I was thinking about to solve these issues here.

G

A

Yeah I mean, of course, like I'm, not sure, what's going to make really the most sense, maybe we don't like this may be something where we put this this these this. This literal include at the end or something or maybe put at the beginning, I'm not really sure we just need to.

A

We know we'll make it make sense for people, so that I mean because at the end of the day, like the biggest goal for this is we want people to be able to copy paste a random crap from the documentation and have it work for whatever they're doing right, and so you know the sooner we can get them to to copy paste in heaven. The better but yeah I. Think I, think this, hopefully would would would would make it clear. But if not you know, that's that's it's whatever.

A

Whatever you think would be good, then so yeah anything else on that er.

G

Yeah about the example example code: will it be the already existing code? Does it have to be the already existing code or you know we can create Aronnax? Oh.

A

Yeah, you can definitely and like the test code, because the test code could be used as a reference for that stuff, but the test code can also get kind of complicated right and so definitely dumb it down to the easiest, make it make it very clear what's going on, and you probably want to have multiple examples for things right, because you know there's different some. Some things can be called differently. For example, in high level there's several ways you can call those functions right.

A

You can even pass a file name and it'll turn the file name into a source and stuff like that. So we would want to have multiple examples for, for all the different call styles for four things so that people could say. Okay like this is the one that I need. Let me grab that example there or like. Oh, this doesn't really you know. One thing may not really fully illustrate. What's going on with one example, usually people require multiple examples to really understand.

A

What's what's going on, okay, yeah cool write this down here so good example, code to size example, model.

A

A

And friends from Tess.

A

Guide then show what those examples basic levels are doing with literal.

A

Include within dioxide I, slash.

A

All right cool anything else on that then or anything in general from you that you want to talk about.

G

Now that's it other than that I yeah. I know you already told us that they'll be redeemed proposals, but I wanted to know on that. If you know the timeline suits all this abstract base classes, the changes.

A

Let's see, what do you mean just because you might you might change your your your drafter.

G

No yeah the draft was submitted before we decided to no.

A

I think you can resubmit drafts. So if you want to change the draft to just mention this, then like, then you could really think you can submit can't you submit multiple drafts, I'm not entirely familiar, but.

B

A

And you can always send me a draft out-of-band and I will treat that. As you know, the latest draft, because I'm tracking the draft separately outside of the Google system as well we've got a separate Google Doc going with the drafts so that we can. We can consolidate comments and stuff and the mentors can all we can make sure we're getting you all the mentors feedback. So not just one of ours.

A

Let's see I guess is everybody else good on here? Is it just Augen who wanted to talk some more about things or did anybody else have anything for the day? Sorry we're a little bit.

B

Yeah I just wanted to do from a certain equal because.

B

A

But I was thinking, remember good, so maybe you like in a certain plan or something because I mean I I.

C

A

Again, could you mute for a second actually, okay, great, so so what I was what I was saying was that you could do a no.

A

This is the wrong PR, but you could do a print basically, so so just run the test so write the test to do load, B and then like don't, write, any assert and then write just say print and then the bytes and then just copy paste that buffer into a you know just copy copy, whatever it prints out, and it should be like what 748 bytes long or something and throw it in an array in the top of the file and then do assert equals on that.

A

It's gonna be a big nasty array, but it will, you know, be a good test right, because if the file data changes anything I'll throw something right, because just asserting that its length, 748 or asserting that there is something in the array may cause issues if, for some reason we start loading the wrong things at some point. So.

B

The Lord we could load the file, the byte buffer, yeah.

A

So basically, let's see where.

A

Yeah so do like.

A

Do like a print, you know, let's call this, let's just say rich by it's equals this. So loaded image equals. Oh wait: butter, yes, utility cool, GTX step load, be image. Bytes.

A

Groups and then we just so it basically just print loaded image and then now copy the output of okay.

B

I, don't like something like hard coded and yeah.

A

That we're well, you basically you'll have hard-coded the bytes of that image into the test file right, which is like. Maybe it's not good or way. Oh, no. You are now I'm, remembering how I did this last time. Oh, you can just take the hash of the image and there's an example of how to do this, because I almost did this one time and then I realized we can just take the hash. So where is it.

A

Grab a schlub, oh great, you know, there's only a million usages of hashlib.

A

It's an attests, oh yeah here, I think it was the idx source yeah. So here's what I did so I D X first last, so this is like basically I looked at the ID X, so you could just you could like print or JSON dump the okay. Yet so look at this test as an example- and this is kind of like this is I- should have commented this more. But basically, what we did here is we went through each of the records like grab, all of them all of the ID X records right and there's.

A

Then we check that there should be 60,000 of them right. So in this case, like you're, only loading one thing right, because this is the source and you're just talking about the one image. So you load it. You assert the length of it is 748 or whatever the link should be right and then you can basically do JSON dumps dotting code and then assert that like and then grab the hash of that right, calculate the hash of that and then print out. The hash.

A

Save the hash is a static variable in the file and then just assert that the hash is equal to whatever your you know, your statically declared thing is after you've done your you're, printing and copy pasting right, because that way, you don't have to paste that 700 length array in there. You just face this ugly hash, which is also ugly, but at least shorter right.

A

So that's it yeah yeah good call on this I knew I completely forgot that I'd run into the tsavorite and in fact it was you and I ran into this before all right. So, okay.

B

I think the jeans great.

A

Great great, thank you all right. So is that all that from everybody you.

B

Know is it cool.

A

All right, well, yeah, so augen did you want. You wanted to hang around talk about distributed, Orchestrator stuff.

D

Before that, like last time, we discussed about the exam, so we still having fun.

D

A

Yes, the locking feature on the dance floor: okay, oh yeah, okay, I had something with this.

A

Yeah yeah yeah, it's probably Arvind I, probably forgot to talk to him. So let's see okay, so for the first one, we've got the calculator example. That's like that's, that's good enough to get people started, and then we talked about the reading. So let's let's go and render II remember what are what are three things. Were we had a good one, so calculator l, configs, shot getter IRC, that's good 100 ping Shawn for rpm file, maintenance ooh that one would be good, yeah, okay, basically, and this one's already up there. Actually, let's see.

A

Oh, don't do giggling kid.

A

Oh yeah right so.

A

Go away all right: where are we I pick? You check out been sick.

A

You guys see my screen wound up here. It's like is this big enough.

A

Here we go. Oh.

A

Wait, oh for everyone who is still on the call. This is something I noticed recently: okay, I screwed this up, but but well I've fixed it just so. Everybody knows that this is weird happening, so hot shame when we switched to doing because you were building the dachshund docker container and the docker container doesn't actually have a user when you're doing it from the development container rather than the production container.

A

So when you do a find and replace on when we do expand user for all of the model directories, let me write this down. Let me make an issue out of this right now: issues.

A

A

Model directory not calling.

A

I flip directories, so we end up when the model directory is.

A

Therese, okay, whatever were switched over to pathway of top path, objects objects. They ended up treating.

A

Squiggly or tilde, as a literal, rather than rather than the home directory because of a lack of a resolve.

A

Call or maybe expand user but I think it's a resolve will sort of do better. I think it does exactly what we want. We need to call resolve.

A

Need to set the directory to property to its resolved self, so so doc, config equals self dot, config dot, replace directory equals self dot. Two or just kind of picked up directory dot, resolved. I. Think I think this is the correct solution.

A

Sorry I just wanted to make a note of this, because I saw this and I just saw it again and I'd forgotten, so labels bug machine learning.

A

Extra short machine learning somewhere, maybe somewhere there, okay, all right I'm, so been sick, though alright. So these operations, what the new is and okay. This is like- maybe maybe maybe not, let's see, okay, so this one can probably go in the main repo, but basically what this was was an operation to take the URL and turn it into a bytes object.

A

So if you give this a URL, it gives you a bytes object and OH uses AI or DP, so this actually may have may need like a separate set of operations like it may need to go in DF, open mental operations net or something because it's third-party dependency, but we could have a regular one within the main repo that that just uses URL, live or something and then declares itself synchronous and being a bummer. No, let's not do that. That's gonna!

A

That's gonna, create problems so, but the question is like how how do we make this? How do we make this? How do we make this? We maybe could do it even easier than this, because it's kind of complicated, basically what we could do is we could pass around some sort of objects. So the thing is that the reason why this this is a good.

A

This is a good example of sort of locking things it's because we have to create this RPM file object itself, and this is a thing that operates on this static file buffer in memory, and so we basically we take the URL. We download the RPM from the URL, and then we just have this array of bytes in memory, and then we take this RPM file object and we say: hey like your rpm.

A

Is this array of bytes in memory rather going to file it on disk and now what we want to do is we want to pass that RPM around and we want to extract a bunch of binaries from it and then, once we figure out yeah, we want to when I figure out what are all the filenames in it and then every time we have a new file. We want to run this. This other operation to say is this: is this binary so was?

A

Was the file a binary and then is the file a position independent executable, which is like a security feature that we have so it's sort of a good example for locking, because this RPM file object can't be used by multiple operations at the same time. So it has to be locked, it's sort of not that great example, because there's there's more complex things going on that might confuse people right like.

D

A

It's got to be real, simple right.

D

A

Right, so that's very, not a good one yeah, so we need to think of something so, and this is maybe just like what could it be? What could it be? Oh you know what would be good is a I wonder how this would work okay, so you pass around an object and if you don't declare a lock on the object, then you increment or you increment, one of the objects properties by one and let's see what do you do?

A

We need just like a classic multi-threaded problem example. So something like okay, we could sleep, let's see what we do. We could sleep for some number of time and then, like add the number of time that we slept for and print it or something it's basically. What you would see is that, like everybody's sleeping for a different amount and I'm, not sure I, guess I'll have to think more about this one, but I think I think well, we can like the main properties of this. One are you're going to have.

A

There will be some, so there will be some.

D

A

Yeah so and I mean usually I think what people do is just something around the idea of like okay, there's there's a bunch of let's see, there's like you know, a bunch of a bunch of timers going off. Oh yeah, yeah, here's what we could do.

A

We could do something where you have an operation that takes like it takes this object that needs to be locked, and then it takes a time that it sleeps for and it takes a number and so after it's left for that time, it sets the object's property, because the object is going to be basically a reference since we're in Python right. So it's all everything's actually a pointer to something right. So it's gonna set that property to whatever that value is and then and then print it right.

A

And so, if you run this, this data flow you'll see a bunch of series of a print statement and they're all gonna like sleep for a different time and then and then print what they set it to and.

A

Yeah well, let's see, let's see, okay, here's we do yeah.

C

A

You you set the property, so you the operation, it's basically this it's.

A

So it's ink or def run me so takes object, it takes time or sleep for and it takes an integer right, and so what it would do is it's a object. . I equals my right and then it does wait.

A

Wait, it's ink, io dots, sleep, sleep! Four and now it says print object, dot. I right, and so it's going to happen.

D

A

What's going to happen, is you're gonna see that, since these things don't all run at the same time, they're all gonna have like different the this is. This is not gonna print. Anything like it's gonna prevent print one value right at the end of print, whatever the last one slept for right, whereas if we go- and we make this so then we'd have to we'd have to sort of examples right like the definition for object, is definition and then lock equals true, and if you did so we'd have one.

A

We basically run this example twice right, we'd run it once.

C

A

Once with the lock equal, true and once without lock equals true, and you would see the difference in the prints and that they print the correct thing right, and I think I think that would illustrate the locking well enough.

A

So let's see- and let me just let's just paste this- it's like we'll just we'll just save this somewhere. So just.

A

Okay, so base basics operations for for locking example,.

A

Okay, you know like this, and that could be like a dictionary. It doesn't really have to be its own object. Okay, so, like an example, I do once with Locke equals or without Locke router.

A

Set in definition, and once it's all right and that will sort of doubt, I think that'll show it.

A

Okay, anything else on that sort of thing there.

D

D

D

A

Yeah, like I, was saying so, if you just if anybody just sends their proposal to me, I'm gonna make sure that all the other mentors see it and then we'll all give you our our feedback and send it back to you. So it's, but you should eventually, obviously submit it through them. Angie.

D

A

Cool sounds good and then yeah anything on the distributed, Orchestrator stuff.

D

D

A

Say sorry now and you're cutting in and out a little bit Matt's.

D

So it's not stream.

E

That's what not stream stream.

D

Okay, great yeah, so if you just use the regular mashed, it's just a fire and 4-bit operation kind of thing, so our data might not deliver. Oh yeah. They have this thing which they call stand which ensures yes,.

A

Driving a water tower no.

D

It will be like, and it's mostly asking where your face like I feel like will be applicable and then share some wants it. Alright,.

A

So they have a lot.

D

A

Stuff yeah, okay, great.

D

Like what what are we going to actually say in between the notes, or is it the orchestrator that, like this matching everything yeah.

A

D

A

Think that the way to do this is probably, let's see, okay, I can't. Let me remember what we talked about here, so we have the notes right, oh and now I, remember: okay. We need to.

A

What we'll do is in a node will run a node will basically be a running operation, implementation network right, and so you you, you create this new we're creating. Instead of a memory operation, implementation Network, we now have Nats operation, implementation, Network and we're gonna, add some sort of new new CLI interface here to say like okay and Stan, she ate one.

A

You know instantiate operation, implementation, Network whatever with this config and load these operations into it right and that will basically be the node CLI command, and so this implementation network is basically going to be subscribed. It'll be subscribed to some channel where it's like. Looking for it'll, probably like a few few things right, like first of all, we're going to need to make sure that it's it's you may you may have something where okay, so let me write this stuff down, so we're going to add a node command to the CLI, which.

A

Instantiates and our presentation network in its case, that's operational implementation network and- and anybody like you guys, don't have to stay on. If you don't want I know, we've talked about all your stuff, so this is just this is just.

A

This is only for your curiosity if you want to stay up and listen to this so so, and what we're talking about here, just just for anybody who wants lesson is, is Agins, think you know of proposing a distributed Orchestrator project, so basically we're gonna be able to run the the or the data flows in a distributed setting. So you could have them on multiple machines and different machines would be you know, running different operations and, of course this is.

A

This will be nice because you can get a lot of things done, especially cpu in terms of things. If you're distributing on two different machines say you had like you know, one data flow where one of the operations is run this you know this model inference, use the trained model. Well, that's gonna, be a CPU intensive operation. You're gonna actually need a lot of course to do that if you're gonna do it in parallel, which means you need to do it on a lot of different machines, which means you need the distributed Orchestrator.

A

So we're going to add this node command to the CLI, which basically, this is what you would run on, one of the one of the machines. That's not like the main. So this is like this is a machine that that's tasked with running some set of operations. So we add this node command, which lets you instantiate, an operation, implementation network and it's config and well, it's gonna figure, sort of implied but and lets us know which operations to instantiate.

A

It's within that network, so this node.

A

Node, 1 or node n is responsible for running operations, X, Y, Z, and so what we now have is like we connect so connect up to Nats using and let you sing that streaming right.

A

Okay, whatever I'll just right now, that's streaming.

A

So the and preparation implementation network.

A

Connects up to Nats streaming and the config may have properties such such as like I'm, not familiar with not streaming, but it may have some sort of like namespace that you're in or we could like do this like I, don't know exactly how we'll do this. We could always take it and say.

D

Something like.

A

Yeah, okay, cool great, so what we could do is they.

D

Have channel okay.

A

Yeah cuz, basically, all we really need is something to wear, is I. Think I think what we need here is is we'll need we'll need a couple levels of things right, we'll need or well actually, where we won't. Really. This isn't how this? How should this go? It was basically well how does these yeah yeah? Well, basically, it won't instantiate them. It'll say that they're allowed to instantiate these ones, so is allowed to run operations XYZ, because the data flows will actually come in and instantiate them via their connection of this network right.

A

So, okay, let me just like come down here, look at level and try something else so we've got, we got to look at it like what happens from the top right. So so first thing that comes in, and so we have a data flow data flow wants to instantiate operations, operation of an operation, implementation, so op amp's right, and so it goes okay, and where is this issue issues.

A

So what we need is goes to the operation, implementation networks and it's this issue.

A

So basically, what we're going to need to do is we're going to need to make this abstraction around just sort of like just exactly like sources where sources wraps a bunch of different sources, we're going to need to wrap the operation implementation or that networks and we're gonna need to have the data flow, be able to talk to multiple networks and that way you'd have sort of like you. Well I. Guess in this game, I mean and don't really need to do this.

A

Way, probably don't knew it's hold off on this. Oh, let's hold off on this. The idea behind this one was basically we could have this other thing where you could be hold it. You could be doing things in memory like you could have the in memory network or you could have the Nats network, but for now we should just focus on just the Nats network. So in this case we're go to the Nats operation, implementation network and.

A

Orchestrator side, and so let's do within organ, so it should be within orchestrate or not the nodes. This is the main like.

A

A

A

The the dataflow wants to instantiate operation implementation, so we go to the Nats operation, implementation, Network and we say instantiate stand she ate operation. You know every instance of each every operation innocence we need the sense we need for this flow right, and so then what we have is you know that would be the instantiate method right that we already really close this. That's the instantiate method, right that we already have.

A

So, if you say instantiate and we're probably going to need we're, probably gonna need two sides to the snaps operation and implementation Network, so net.

A

What would watch we call this primary operation, implementation, Network so and then that's node right, because when this this thing will have, let's see well it.

A

Basically, when you call the.

D

A

So when we call the instantiate method right, so let's look at that. Let's look at the dataflow code here so.

A

All right, so, when we look at the memory, defin stanch rollin initiate right. So when we, when we look at this method here, it takes the operation and it takes you know, maybe the operation implementation in this case it will not write good, it has to be able to load it, and so we're calling this method and now right now you have to think about we're calling this method. Basically, we have.

A

We have two situations here right, so we call this method, and this is in the Nats client, that's running on the primary primary node or the like the primary instance right, and this is where the orchestrator is running right, and so, when we call this instantiate method right when the orchestrator calls instantiates on the on the NAB's operation implementation network, it's taking its really needs to publish this to some sort of queue right and then all of the nodes are going to be listening to that queue and when they, when they hear when they hear that they are.

A

You know that we requested an instantiation of this thing. We'll have to do you know some sort of like the TCP three-way handshake and say like okay, I'm, going to instantiate this thing and then the orchestrator says: okay, like you can instantiate this thing and then we say: okay, I've been stanchion at this thing right and then we know that we got one that was instantiated right. So we know that one of the nodes successfully instantiating an object right or an implementation right.

A

Does that make sense, so you've got two sides, basically on think of it as like, so on. The server will say server and client in this case right.

A

So the server is basically the the master instance right, and so the master instance says that has the data flow says: okay, I'm, sending out I connect up to Nats and I connect to the some sort of channel that that all of the nodes are listening on to understand when they need to instantiate objects right, and so, if you, if you so so, all right, sorry, okay, so so main server connects to channel that nodes are listening on to determine when they need to instantiate things right and it gets the data, and it goes in instantiates every instance, within every instance of every operation right and so to do that.

A

It calls this instantiate method and server-side. This is going to publish to that queue and or publish that channel, and then the nodes go a I'm allowed. You know this is in my allow list of operations that I'm allowed to instantiate. Let me send back a message, so, let's let me write this down to go so remember. Essentially, every option or so so publish need, for instance, to the channel which nodes listen to so nodes, see incoming request to instantiate an operation they within their allow list.

A

Published in the request see incoming request instantiate an arbitration in their allow list.

A

A

And then they respond that they will or that so they instantiate she ate and respond that it's it's been instantiated.

A

Alright, let's and she ate the requested.

A

Operation and then respond that it's been instantiated so now and what would really have helped here is if we made we started making a flow diagram, but so so now we've completed that initial flow. Where all of the operations within the data flow are. They exist on one of these nodes right there, implementation has been loaded into memory and it's sitting there right. The contexts have been entered right. The main context has been entered on. All of them right is that does that make sense? Okay,.

G

A

A

Okay, so yeah, so we we have the main instance and then all of the nodes for whatever ones that they're allowed to instantiate they've now instantiate at them. For that, for some data flow right and then and and what that involves right, is going to be the serialization of the operation structure right and and you could, you can choose to serialize and deserialize that, however, you want right like I, don't I would assume that's sort of like you could always just wrote JSON over it right so.

F

A

At a minimum you basically- and this is this- is so this is exactly why all of this stuff had to be built to be to be serializable and why the input parameter right when we were talking about validation. Now, that's that's. When I started to realize Oh like we made that a function ohm and that won't work now so.

D

A

Instance, name will be fine, it's just that now it's sort of found to be bound to the data flow right, and so it like you know, instance name- is the data flow thing and operations. Don't really have a constant of instance name until they become an operation in a data flow, but that's the separate thing so now we're at the stage where we've got the master node and we've got all the all the other nodes have the operations that they need.

A

So now we basically we wait for we wait for inputs right and or what we see. We see all the inputs to the network right and what we're gonna do here is. We should probably have like the flow, so the low data flow dot flow tells us where all of the it tells us where all the inputs for each operation can come from, and so what we need to do is basically have something that so we, the orchestrator, will decide the permutations right and okay. Let me let's: let's look, looks like it.

A

Let's look at this diagram, this handy-dandy diagram which I created recently, so it's okay. So what we just did here was the operation implementations register with the operation implementation network right.

A

So now we're looking at okay like there's new inputs right so with those new inputs, they come into the input network and, let's see, is this a case. I think we were saying that we might want to do.

A

Let's see we might want to do.

A

Let's see, let's see, let's see, I think at a minimum right now. We don't need to worry about doing this in Nats, I think what we can do so, basically the input network. So we we have a few options here, right and I. Think I think what we're do first is go with the simplest option which is going to be. We only run the optimum operation, implementation networks it over Nats, basically right so anytime, we decide that there is new inputs.

A

Basically, everything has to be on in this case, like all all the new inputs are come like things that aren't the outputs of operations. They have to come through the orchestrator right where ruby orchestrators living at this point like for round one here right and then, as you advance further in this project, you might do something where you implement the input network on top of naps as well. Right so and sorry, this is I'm making this more computing.

A

So basically what we could, what we could do here as a first start, is we look at the it just sorry? Let me remove this, because this is going to make it like we're getting too into the weeds, we're you can. You could take.

A

Let's see, let's see, I'm sorry I'm, sorry I feel like we figure this out and I might be strained from what we said last time. So I want to make sure I'm consistent here.

A

So, let's see yeah, there's Network contracts, so basically every time you want to run an operation, you would send the parameters out to the queue out to the channel that that operation, so the implementation network every time an extension it's an operation. It will now have something: that's listening on a queue for that operation to receive a new parameter set for that operation right.

A

So, if I publish a parameter set to the queue I I run that I run the operation right using using that parameter set and then I publish the results back to the back to the to the C I, publish the results back to the input input set right, I believe that's what we do now. Let's see yeah I think this is. This is why we decided that.

A

Yeah, okay! Yes, this is what happened to Brenda dispatch yeah. Okay, so does this guy take a.

A

Way to do yeah no CTX, I CTX add memory. Input set all right, okay, so that's the deal so we do need to add. This is why we said this last time, so we were talking about last time. Creating we'd need an operation implementation network, that's based on Nats, so that we can listen for new parameter sets and then we're also going to need an input set or sorry, not an input set, but a an input.

A

What is it input yeah a input network based on Nats, because when we do when we, when we run the operation here right and we take the the outputs and we convert them into new inputs, and then we take that those those outputs that are now new inputs- and we add them to the input set context or yeah. They input the input network context, so you're throwing.

D

A

A

So the parameter sets get so when we, when we're, let's see when we're, let's see what was it, what was it.

A

Oh yeah, okay, we have so the orchestra sits there and it waits for the input set right. Okay, you know yeah, we have to implement all of these, so the the input network is responsible for feeding all of the inputs in right. So you need you need to have a Nats channel for new inputs of different types right and I'm. Sorry, I'm not doing a good job of explaining this right now I feel like I, was doing a better job. Last time.

D

A

We didn't get into this detail yeah, so, okay, okay,.

A

How do I explain this? Okay.

A

So, every time a new input enters the network, the orchestrator is going to look at it and decide if it needs to if it needs to dispatch if it needs to create a new parameter set for it right to dispatch an operation right and so we'll have the main node, which is going to be to be deciding whether this is the master node to sides and discipline and sends out parameter sets right. So it needs a input network. That's going to feed it. The results that are published from whenever these operations are run right.

A

So there are other nodes running the operations and when they run the operation they it call when they run the operation, they say input, Network, dot, add right, and so what we need is that input network that we're adding it to to actually be something that publishes all of those inputs, as as an input set yeah to the to some Nats channel that the master node. It's listening right, okay thinking! So so, let's see, let me write that down. um So, let's see okay so with or goes to the primary napped operational notation, Network.

A

Okay, when the intention, so when so, let's see okay, so the first thing is Orchestrator. So, let's see to run the dataflow, we add all of the inputs inputs to the input, Network right and then let's see and then when up when things are added the input Network.

A

Yeah we look at their context and and then we dispatched the appropriate ones. Let's see, yeah operation parameter set pairs, yeah, okay, to run the data flow. We add all the inputs, the input network and then we generate a parameter set set. There's.

A

Never two pairs and then we say so and for each parameter set or set pair or let's see, operation pair, we call dispatch right. So we call.

A

I said yeah and CTX is the operation implementation networks that we call dot dispatch.

A

Which sends out that operation or sends out that parameter set to the channel we're nodes will be waiting to be repaired. We call this Nats operation, Network dispatch, which sends out that perimeter said to the to a channel.

A

Associated aidid with that operation, where nodes who have that operation operation instantiated, it will be listening or will be waiting for parameter sets so that they again or yeah we'll be waiting for another said once they get one. They run it and add the results back to the Nats, no operation or input network.

A

So the nets that's node. So and then we say the Nats node input, Network dot, add publishes the input set to a channel associate. Ok, and this is where the flow comes in right. So we look at that flow and this channel every time.

A

Every time we have so, let's see so, do we just publish it and say we can just say where it came from yeah we can just publish all the inputs as an input set and then we'll filter through the origins when, when we're, basically listening to that channel, so the Nats note input inputs that add publishes the input set to a channel for that data flow.

A

Then we do so. That's that's mm, let's see, like master input, network dot added and like I'm, calling them node and master right now, but they may very well end up being the same class I'm, not I'm, not sure so.

A

Dot added right, which is what tells us, is waiting on that channel to see new inputs from operations come in or new inputs.

A

Ci new inputs that were the outputs of operation implementation networks come in right and then basically, the cycle repeats I. Think I, think that will that will do it at this one on, let's see because, let's see what we've got, let's see first parameter sub: here's your redundancy, checker, okay, The, Dispatch, I- think we might need the locking network to let's see yeah the redundancy checker is gonna, be up T, let's see!

A

Well, we just move this stuff in here, so redundancy checker, but don't think we'll need to implement one yet like just to get this warning right because we're basically we're concerned with here, there's like how do we just get this a bare minimum example of like what is it like distributed working right?

A

That's really the project here so I think if our CTX doesn't get used, I think our CTX just gets used.

A

It's only getting used within the orchestrator or wait. Does it get used, gather inputs.

A

Yeah, a memory input sick now working on their. It gives basically anything that gets used outside of the orchestrator wall. That need to be implemented outside of the orchestrator, but gather inputs is something that's only gonna be called on the orchestrator side of things. So yeah you may not need to implement the RCT X in it in anything other. You can probably reuse memory RCT X, because if it's only being called on the orchestrator side of things, then obviously the the nodes don't even need to need to know about the redundancy checks right.

A

They just need to go, execute the operations so I. Think. If you implement the input Network and the operation implementation Network, it will work qgi you, basically you publish out things. They listen for. What's published, they publish out results, you listen for the results and you do the standard lock network, we're done in T, Network, redundancy checker on the orchestrator side and then yeah, you publish yeah you just published out for Andrew setters.

B

A

I think this is I think this is what's required here to get it to get it working.

A

Let's see yeah I think that's it. Do you see anything let's write here.

D

A

A

Well, let's see I mean.

D

C

A

Be detailed and I have to understand like what your timeline is other than so there's there's a couple situations here. It depends on the proposal right in the end, the project that you're talking about right, because with some things, it's going to be very clearly defined. Right, like it's gonna, be obvious when.

E

A

And it's going to be like for the example, the tutorials I think.

G

You said you were planning.

A

On quitting, probably the tutorials in your proposal right, it's gonna, be very. The most important thing with the proposals is the amount of time that you spent working on it and that it's clear that you're going to spend like as Google is expecting that people are spending 30 plus hours a week on on their project right.

A

So it needs to be clear that, like the amount of work that you set out to do is something that you could that you would be spending that much time on and then it also like, like it's also just needs to be clear to the mentors like what you're planning on doing right, like obviously I think everybody that's on the call right now, it's very it's very clear to me and everybody else what you plan on doing right like there's. No, there's no question about that. It's really just more of like okay.

A

How do you plan to do that within that that, like that time period, right like how? What are you going to be working on? You know each week so that you are actually like making that time, commitment and in this case, okay. So in your case right, your your tutorials I think you'll be able to fairly easy estimate like how long that.

E

A

Right and with this it's sort of like okay, none of none of us, really know how long this is going to take you right.

D

A

And that's when it when it comes to things like that, it's like! So!

A

If you're acting like it's gonna, come down to like you need to set some goals for yourself, like okay I'm, going to have the first of all you're going to need to do things like and- and it depends like, if you just jump right into this or if you keep hitting issues where you're going to need to do things like okay, go play with Nats for a while right and you're going to need to just write some code that works and Nats that works with now it's using their api's right and that's going to be something that you're gonna need to budget time for so like it might.

A

Take you a week to figure out okay, how do I write a few things that are published and subscribe to different channels and talking to each other appropriately right and then now? How do I write that with indie FML right and now? How do I figure out how to write the CLI command that so break it down into into the various things that need to happen here right? So we know that we need. We know that we need to networks to new networks right.

A

We know we need the operation implementation network, that that is, that is listening for things, and then we know that we need the input network right. So we know we need a network, that's published and subscribed.

A

Sorry, obviously um we need yeah, we need it. We need each network, so we need to write that so you need to first, you need to figure out how to do channels, and then you need to you need to figure out how to do the implementation network, and so we've gone through here, like okay I needed to figure out how to instantiate everything I need to make sure that the instantiation process is working right and then you need to be writing all the tests.

A

For all these things right and and a lot of what your workflow I think is going to look like is you're gonna want to leverage heavily, and let me write this in here so going to want to heavily leverage the unit test, dot, mock dot, patch functionality, because you'll want to write the the nats.

A

Operation network, size and input network and put network and test like and just the like, you you'll, want you'll want to set it up so that you can basically fake the Nats connection and not actually have them, connect it to Nats right so you'll want to write the operation, implementation network and the net input network and have them doing like and have them not really connected to Nats and and so you're gonna need to figure out how you're gonna set up all that testing stuff right.

A

So so, basically, when we say okay, we're spinning up like this example use case that we just talked about where we've got them the masternode and then we've got all the other nodes. So, for example, just that that base instantiation use case right where you've written the Nats class and just to get it started. You say every time it says instantiate right, because we will go through and we instantiate everything. So every time it says, instantiate there's some call to the Nats API in there that's doing the published command.

A

So you mock out that public, and you said yes, it was successful right and you make sure that everything worked right. The rest of your code work, even though you weren't really connected to Nats right, so you're gonna need to figure out like you're gonna need to budget time for doing all of that and I think you'll sort of find it quickly here that you've already got like just within the things we've talked about here. You've got like a few weeks of things, need.

B

A

You've got like you know, maybe a week or so, for each like it's probably like a week for each tutorial to make sure that it's really all clean. And then it's like a week to figure out Nats and then it's a week to write the basic like what you think the implementation network should look like. And then it's probably.

E

A

Week to really test that thing and then input network week to test and then the rest of the weeks, you're figuring out how to integrate it correctly right.

G

A

Then write the docs you'll need to write some docs for it and I'm not sure how many weeks we have exactly it's like 11 or 12, or.

F

A

F

A

Think you can safely say, like you, can safely say that that, like you, you can you can budget some weeks and then you know with a project like this works.

E

A

Yeah yeah and like the other thing, is like when accepting proposals like like you know, we know we know like we know you and I know I think we know most of you guys on phone call and stuff, and you guys are you guys, have been here and you guys have been working right and so it's like you know the people who we know we can trust. Actually like do stuff. Those are above I know there are some people.

A

There are always people that submit that we've never heard from before right and it's like well I, don't like no, we don't know that you're actually gonna follow through on this, because you haven't actually done done anything before right, where it's like all. You know the people who are existing members of the community. It's like, oh, we know, we know where we are. We know we can count on you to actually do this right.

A

So if you say like I'm, not sure how much time this will take like well, we trust you that you will be working on it because I'm going to be watching your poll requests and if you're not doing it like I'm, going to be expecting that the poll requests are coming through with the work-in-progress tag right, whereas other people- they you know they they haven't, submitted anything and they don't know that they should be. You know putting up their work with a work-in-progress tag so that we know that they're doing something right like we're.

A

Seeing you actively push work when you have a work-in-progress tour request. Oh right, like we see the lines of code coming through, so we know you're working on it, even though you haven't gotten it merged right, and so that's important right.

A

But yeah does that sort of.

G

A

All right anything else for anything.

A

Well, if that is it have a good night and I will talk to you guys next week and I'll try to clean these things up, and let me just recap for myself: there's Conda and there is yes, the the Dement or portal stuff, and then we need some issues to track this. So yeah cool all right! Thank you guys and have a great night and stay healthy out there or well stay healthy inside because don't go out anyway.

F

A

You guys, but thanks.