DFFML Weekly Sync, 2 Mar 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Weekly Sync 2021-03-02

Description

Meeting Minutes: https://docs.google.com/document/d/16u9Tev3O0CcUDe2nfikHmrO3Xnd4ASJ45myFgQLpvzM/edit#heading=h.kubpttcjvycn

A

Hi john hi, everyone.

A

Doing good, how about you.

B

B

All right, okay,.

C

All right, so I I was saying all my computers are out of disk space, so this could be interesting.

C

C

Weird all right: okay, we've got everything on here: let's bring up the medium.

C

C

So in attach, did you see? Did you see what was going on with that um the h2o model.

D

Yeah, I think you have merged that. That's true right.

C

I commented back, and so I did that a little bit that was that was. I was too eager to merge that. So, let's see um there were some issues with it. I merged it. Basically, it wasn't enabled in the ci I merged it found out it wasn't able to build in the ci. Some issues happened. um Basically you you need to reopen a pull request on that one because it was I wasn't able to to. I had to revert it um because the ci, the tests weren't passing in the ci so.

D

Okay, yeah, you have commented here.

D

To reopen the pr.

C

Yeah so- and I think the main issue here was was both of those I hadn't, I hadn't. I thought that it was up to date with master when I merged it and it wasn't, and so I then we had issues um so yeah, we'll just we'll just have to redo it and make sure it's up to date with master, and I think you'll you'll see you'll see what the what the ci issue was.

C

um Okay, let's.

E

C

C

All right, yeah, I'm sorry honcho. I just went to look at your pr last night, but then I had to I had to do some other stuff. I've got a it was.

C

I had. I have this meeting that I run this morning that I had to had to do some stuff on, um so I haven't gotten to that yet, but I will I will review it. I will probably review it after this meeting um because that we need to get that through. um I think that's looking, you know very, very close to done there. So all.

F

Right, oh yeah, sure that's fine.

C

C

All right, so what do we got today? Natash? uh What do you got going.

D

um Actually, I wanted to know about the http uh thing to run the python example: okay,.

D

Right there is an issue.

C

ah Is this the python client example.

D

C

Yeah yeah figures.

D

Yeah in a service http, uh the python example needs to update it. The issue.

F

D

963, I think yeah here this this code.

D

So how to run this- and there are some problems in this particular course so.

C

Yeah, oh there are, these are just straight up that will not work.

C

Okay, yeah, yeah, okay, great so, and I think we had an issue yeah to go fix this. So.

C

You know why was this so, oh, this one wasn't confusing: it was the javascript one that was the one that was um that was uh going to be more difficult. um Okay,.

C

Okay yeah, this should not be. This should be pretty straightforward for us to make this. This testable um it's resource. I think client examples.

C

Okay, uh anything else on your in there.

D

um About the unit test.mock in my previous pr, I mean how to write a test for that cool.

C

Yeah that we can go with that.

D

I have read this uh a lot more thing about this unit, dot mark but uh then cleared what what we are doing and.

C

D

How can we do that.

C

That's a good one to go over as as as a team here, because that uh it's it's it. It's definitely can be tricky um the first time, so all right uh anything else there or that's right. No, no! No!

C

Okay, um so uh hashim.

A

Hey yeah, can you hear me yep.

C

Hello, yes, we can hear you. Can you hear me.

A

Yeah, okay, so I've made some progress on the confidence issue uh separating out the confidence.

C

A

From with the predictions here, so I've just started to uh deal with the sources uh in respect to that issue. All right.

C

A

But uh I'm not uh really that sure uh how uh how to deal with it. Okay,.

C

Okay, uh anything else on your in there.

A

uh No, that's it.

C

All right, great saksham.

G

uh Yeah, I'm just here to see what everyone's been up to great.

C

Great um and we let's see we we figured out when we last talked, um did we have a path forward. We had a path forward right.

G

uh Yeah for the config stuff yeah, I.

C

Haven't touched it yet all right just have last time, yeah. Okay, sounds good great. um uh Let me just write that in the notes that we, so we have a path forward on the config stuff did I send you did I send you the recording that we did of the last little bit of that?

C

I can't remember.

B

I think I think not.

C

B

C

um Okay, all right, let me just check.

B

G

I know you didn't send there.

B

G

Oh, oh, okay! Yes, you added it in there.

C

Oh, I added it to the dock. Okay, great great great. I swear to put it somewhere all right, okay, um all right and then sudhanshu is there. uh So is there anything else that you wanted to talk about um or just just see.

F

uh Yeah I had I like, like now right now, like I'm trying to uh like understand about data flow okay, so I have actually taken up an issue like I want to work on this issue. Number okay, uh 660.

C

F

So, like I had to discuss about it,.

C

Okay, cool all right anything I missed on the agenda here from anyone.

E

All right, great.

C

C

C

All right, so, um let's see, let's look at this um okay, um this unit test mock. Oh great, that's right! Okay! I have no. um I have no disk space. Okay, um let's see yeah. Let's, let's do this one! So why don't you pop open a terminal? And we can? uh We can? You know, walk through this natash.

B

D

D

H

H

H

D

Next, I just forgot the name.

D

So the file is in the fml.

D

D

This is the file right.

C

Yes, this looks like.

F

C

Yeah where's the cash.

E

E

C

Or wait is this? No, this is test. No, this is uh we need to be in.

H

C

D

H

H

Okay, I should.

H

C

And let's can you pull up the patch side by side, so everybody can see what we're testing here.

C

The the um yeah or, I guess your changes so just.

D

Dfmlutil.Net again, I need to open it.

C

uh You know the file that that has the implementation, so dfml slash util net dot, py.

C

Sorry, vi: dfml, slash; okay, you need to go up um up to the root dot, dot, dot, yeah yeah, okay, now vi, dfml, slash util net.

C

Okay, so all right great, so let's go down to where you did the.

C

C

All right, so, just as a recap.

H

C

Everyone, so what natasha was doing here was we. uh We had an issue where, basically, um if there's any sort of failure on the unpack, so we download the file we validate the contents, we extract it um and, let's see: okay, not directory used or uh maker extract, download path, directory path, okay, cool, um so yeah.

C

If if basically we download the contents of this archive, we extract it, we create the directory, we distract extract it into the directory um and then once once, we've done that um we yeah we extract it and then then the problem here is: if there's an error on extraction, then the path still exists and we ended up in this state where we didn't. You know that you end up with this empty directory um and so that that was just you know. We don't want that right.

C

Then then we think that there's a directory and there's not and there's no contents in there. So the the goal of this patch, this pr is to um you know, remove remove the directory if it doesn't exist or if, if there's no, if we fail to extract so there's no contents in there. um So we have this. Try accept block, um and then we do this arm tree and there's this little error handler in there that something about windows. What was the background on that?

C

E

To do with it's not.

C

Writable for some reason, and so you make it writable and then you try again.

C

Yes, that's what's going on here, so basically you said that was there something about windows or something? I can't remember what you said, but basically yeah.

H

Yeah, it's it's windows, error or something like permission denied right, okay, yeah, and so basically you have these read-only.

C

Files for some reason, and then uh they get created read-only, and so you have to ch mod them to be writable, and then you can. Then you can continue the the removal, um so uh we don't need that else. Block there, um just just that's something that we should get rid of else pass. That's a you know the nothing block, yes right there, um so, let's great um all right! So now now what I was saying here was: okay: let's try to test this rm tree thing.

C

You know to make sure that there's no, that there's no um or let's, let's, let's test this thing that we did right. So we want unpack archive to fail so that arm tree gets called right and that the directory is now no longer there right. So this is this is the test case we're trying to write right now.

C

So let's flip over that that test file.

C

Okay and let's copy paste that that function at the end that test cache download unpack archive. So this is the you know this. This verifies that we extracted. um You know this. This tries to it, creates a download, um and uh so can you copy paste. This block.

H

This last one can wait yeah it copy paste that last function and then we'll rename.

C

It to um failure or something you know, test cash, download, unpack archive failure is what we'll call it or something something like that. You.

D

Know um it's done. I think.

C

Yeah, all right, so can you rename it to have like a underscore failure at the end or.

C

C

All right great, so um there's a few things going on here. Obviously, um but what we're most concerned about is the um is is okay, so let me just give somebody for explanation yeah, so so we are actually going to. uh We don't need to worry about verify, extracted contents because there's not going to be any extracted contents.

C

um So what we do need to do is we do need to await funk, um and so so, let's, let's think, let's just think about the flow here. So so the existing code sets up this test server and if you go up to the top of the file, you'll see that the test server basically just creates this archive.tar.gz um and it creates it with the contents that this verify extracted contents function, uh expects um so and you'll see that that yeah, so that verify extracted contents. Basically, this thing it creates a tar file in memory.

C

um This verify extracted content, says: okay, you know I expect there to be a directory. I expect there to be a file that says hello in that directory. It says world and I expect there to be a binary file. That says, you know dead and it has beef in it, um and so uh let's see yes so that that's and then, if you look into the the function, that is the http server handler you'll see how it creates an archive in memory um that that's not really important for this.

C

So the point here is that we want to make sure that this thing blows up when we try to unpack the archive. So what we want to do is we want to have. We want to use the unittest.mock library to have the unpack archive function. Unit test.mock can change things under where you use it in a width block and then anything that happens under that width block you're modifying the behavior of the function that you asked unit test.mock to change.

C

So in this case we want um in this case we want uh the unpack archive function to raise an exception right, because that will trigger our rm tree block, and then we can verify that you know there is no directory path right, um so does that make sense?

C

Yep, okay! So let's let's go into our new function here.

B

C

Yeah, okay, so uh so async funk extracted return, extracted. Okay, great um so, and that's just our dummy little test handler func function right so usually we we usually we use this decorator on top of some kind of function to provide it with. You know the extracted archive um and we are not going. You know we, we don't we. This thing is never going to get called in this case, because you know it's going to blow up, um so we can actually go on that funk, so async def funk on line 119 that return extracted.

C

Let's actually just make that pass.

D

Instead of return rate, yeah.

C

And let's see where this are, I think we want our no curve.

C

Let's see where's the coverage from yeah, so let's say um oops so now, let's put a comment on the same line as pass, and this says so: do uh it do insert mode so do pass space comma.

C

And then uh sorry, not comma, um uh uh pound sign and then space pragma, colon, no space cub, and if you look at the dot coverage rc file, I'm not gonna make this open it right. Now uh sorry, pragma call and space no space, cub cov.

C

All right great! So now this, if you look in the doc coverage, rfc file, you'll notice that um there should be space there, yeah thanks all right.

C

So if you look in the coverage rc file, we don't have to look now but you'll notice that if it has a ignore list- and it sees this comment on a line- it's going to ignore this line, and so since we're never going to call the we're, never going to call this function because it's going to throw an error, we want to put pragma no cuv here um because or else coverage is going to be like. Oh, why didn't you call this?

C

You know you should be covering this line and where we don't want it to right, it's never going to. uh We just have to define the function so that we have something for the for the cache download output. I've decorated a call okay. So let's go to the bottom of this function.

E

C

uh Are we at the very bottom, okay, great? So um all right, so we need to call funk right, um but we're gonna call funkin uh well, let's first first actually yeah, let's call funk first, so try doing an await funk.

C

Okay and below a weight funk, do let's see so where's that verify extracted content so check out that verify extracted contents function.

B

C

All right so yeah, it says assert true is der right, so we want to do. We want an assert, true block and okay, we're going to do extracted and we're going to. You know, check check if this thing is to directory, but we're just going to check for extracted. So um we can. We can uh we let's go all the way down right, so we want to basically do the opposite of that right. We want to make sure that this thing is not there right when we're done with this function.

C

So we want to do a assert false. You know self.cert false.

C

And then we want to do that archive. So it's that pathway.path tempter, slash archive.

C

Just yeah all right: well, okay, wait! No go all the way down! It's the third argument to cash download on pack archive, I didn't know it's the third.

E

Argument is it.

C

No, not that one, it's the third, uh no all the way down go all the way down.

C

It's the third argument to cash download unpack archive on line 115., okay, yeah right, so we want to make sure that this thing doesn't exist right. This is this. Is this? Is our test case right? We call the function. We make sure this directory does not exist, because, right now this directory will exist. You know before if we ran this before your patch, this directory would exist right. This is the bug that we're fixing, so, let's copy you know copy that and put it in that assert false.

C

C

So yeah you want to assert false, and then you want that yeah great.

D

So expected: oh no! This should.

H

C

Let's see yes, great yeah yep, that is exactly what we want. Okay. So now, if we try to do this, let's say so: we, okay! So in this current situation, obviously this is going to uh fail because that's going to exist right. So what we want to do is we want to add a little. We want to add two width blocks around a weight func. The first width block we want to add is the uh it doesn't really matter what order we add them in, but I'm going to.

G

Do this with unit.

C

Test dot mock, um see.

C

Here: okay, let me pull up the docs, because I want to show you guys the whole process of how to do this, because this is a really important way to write tests. um Okay,.

E

I look this up every time.

C

There's a patch yeah, I think it's side.

C

Effects all right, yeah, all right, great um mark: okay, let's pull open the documentation for unit test. um Where is it yeah unit test, dot, mark yeah.

C

Okay, yeah all right, so there's this patch function that basically says all right. This is this is the thing that we're mocking right, so we're going to import we'll make sure that this is imported, and if you go down here a little bit, let's see go down a little bit more all right!

C

You can see yeah, so this sync, this syntax here so with patch the object, you know and say the return value will be one or none right, and this is the kind of thing that that we're doing here, so not that it's not the uh the one above that is really the one. That's that's more interesting to us the example about that where we have that with patch.object product production, class method, return value. This is the type of thing that we're interested here when we're using unit test mock.

C

You know for the most part um and there's lots of examples for the test right, but but this is something that we care about so yeah. Why don't we copy that and we'll flip over back to the tests.

C

C

So you add that with block there yeah we already have unit test.mock, so we don't need to import that. We just need that width.

C

C

All right great so now we want to patch the, and so we don't have to have it imported. So there's two ways you can do this: you can do it on something you've imported, which is the way that it works right now. Patch.Object or you can do it on something, that's just a string. So we just need to change this to width unit test.mock.patch and then get rid of that dot.

C

C

And it's unit test.mock dot.

D

D

C

uh Get rid of the dot object there, because the dot object says I'm doing this on a you know, on an instance of something and that production class is an instance of production class all right. So what we want to do here is we want to do shtutil.unpackarchive where method is so in that string method right there so method. The string that is method right now will be the first argument and the contents of the string will be shutil, dot, unpack archive, and that says to unit test mock patch.

C

It says I want you to patch the shutil.unpackarchive.

C

C

And we're going to make it raise an error and then we'll see when we run this test, that it's going to blow up.

D

So the the first parameter is a function name right.

C

The first parameter is a string that is the name of the function and the including the module, the full path, so shutil dot, unpack archive.

C

D

So the function name is.

C

It's it's well, so it's shutil dot, unpack archive, because that is the function that we want to make raise an exception right so that we can trigger that. Try accept.

F

C

Is anybody not following what we're doing here? It's okay, if anyone is not following, does anybody have any questions right now.

C

Okay, um so yeah, so then remove that method and remove the return value equals none right, because we don't want to set the return value all right. So right now we have some. We have a call that says: okay, hey patch sh util, dot, unpack archive and make it some kind of mock thing right and basically mock the mocku library will go through and and do some tricks and and sort of make it.

C

So you can call this thing and it won't do anything right, um and so what we wanted to do, though, is we wanted to raise a value, so we can pass and if we go over the documentation you'll find that, but it might it's not, let's see so there's if you go to the documentation you'll. Look there see if you scroll down, you'll, see return value and under return value, you'll, see, side effect and side effect is an exception that it's going to raise.

C

um So we can say side effect, yeah perfect, so we can say side effect, equals exception and then it'll raise an exception. um Yeah and if you scroll down further you'll, see it exactly so usually usually things like return value, the most helpful here and there's some there's some.

C

I think the most elaborate examples within our test cases are yeah, so copy paste. That and- and let's see what see, see watch what happens here.

C

The most elaborate examples of the usages of mock that we have, um I believe, are um are uh are in tests service to test dev site effect, equals exception, great yeah, all right so now, and you don't need that as moth mock method in this instance, because we're not going to actually do anything to that um so that that's more useful. You know, if you're trying to make sure, like the other examples that are in there, that it's called or something so, let's indent that await funk and let's watch this thing. Explode.

C

Because that's this is the way. This is just a common way that I like to write these tests. Is I like to watch the test fail first right, because then I know I know. I know that the test is passing on purpose. If I see the test fail once right, okay, so let's indent this block here and then let's run the.

A

C

You can open a new terminal or something.

C

C

C

I think you can pass dash k to it and it will um yeah there you go boom, blew up perfect all right and where is that so in yes, it went you see that that third line up and it said unpack archive and then it blew up and it raised the exception. So that's exactly what we wanted it's failing now right. So um oh wait! A minute. Okay, okay, extractor call raise exception, um yeah, okay. I think this is what we want.

C

um Let's, let's find out, let's go, let's go finish the test case, um I'm not seeing it raised from the other exception, which is what what it should have happened there. um But I think this is the test file.

C

All right, okay, so now all we want to do is we want to say okay, so so now we want to add a little block that says with assert raises.

C

So within this block now we want to say: okay, we're raising the exception. Now that exception that we raised should trigger a further exception that says directory fail to extract error right, so we want to make sure that that exception's getting raised so now we want to enter with block here. That says, with assert self that assert raises.

C

Whatever your error is self dot assert raises.

C

Raises with an s at the end,.

C

Wait so sorry there's a there's. It's assert raises r a I s. Yeah there you go and then so. What is that exception that you raise right because you're you're, you know as there's? No as here it's just just call the function. Yeah indirectly not found yet directory.

H

C

Founder exactly yeah, so we want to make sure that that thing yeah directory, not extracted error. We want to make sure that this thing's getting raised right.

A

C

And then we indent that right and then we make sure that hey no, we don't, but we don't indent that block right, because we want to make sure that we call the function. It can't extract the directory and then once it couldn't extract the directory the arc, the you know, the the place. Isn't there. So in indent, funk, don't indent the other one.

C

All right, okay and then make that a series as a function call that passes directory, not extracted error and then we've successfully. This is how we write a unit test mock test.

D

What what need to change.

C

This is assert raises, is a function, call.

H

Let's rotate this okay, it's a function, yeah.

C

It's no you're, passing you're passing directory, not extracted error. That's the first argument of assert raises it's just like assert, true or cert false. It's a function.

B

C

Yes and pass past directory, not extracted error as an argument to assert.

E

C

Just delete delete the end, close paren and the period and put it at the end.

C

C

Just it just close paren there and replace the dot, you have okay, all right. Well, okay, you just put a close parenthesis here, just to close parentheses, all right and now replace the dot. That's immediately before directory, not extractor. Okay run the test great all right- and this is our completed example and then now we'll move on. um So this is how we wrote. This is how we write a unit test mock test, and this is so. This should be correct.

C

If this is not working, then then we need to look at this further, but we'll look at it: offline, yeah, okay,.

F

C

You'll need to import this right, but this is how we do it just for reference um and now now we all know how to do that, because I think we have not been writing enough tests. So I wanted to make sure that we we cover that completely all right, um great cool.

C

So so I'm going to let you work with that offline um and then and then, and we can circle, you can ping me offline if you still, if you still want to need more help with that, um but otherwise we're going to move on now um so um and then the http service example. I will cover that uh offline, because that was something that was on my list of to do's anyways, um all right.

C

So, let's see uh so hashem. What is the um what's? What's the issue um with the confidence breakup stuff that you're currently on.

D

Okay, thanks thanks. Thank you.

C

H

A

uh I've been able to speculate uh confidence from fiction and uh handle most of the tests, uh this server failing afterwards.

A

As a result, uh I've just started to look into the tests that are failing related to the sources and uh I think it would really help if uh yeah you give me a lead as to how it might be impacting the sources and what need to be.

E

Fixed cool, um you do have a branch, that's up to date, that we can pull down and look at.

C

Do you have a branch that we can look at.

A

um I haven't uh pushed the most recent changes, okay, uh but uh I think uh you can still pull that uh the one from my pr.

C

Okay, let's see I haven't seen the pr.

A

C

So, okay, so this has this the source issues in question: yeah, yeah, okay, great! So let's pull this down yeah.

C

Oh god, oh my god! All of these machines are okay. Oh no! Everyone just saw my password. Okay, god, damn it now. I gotta strip that from the recording change, my password god, damn it all right: okay, um great good job, good job, john! You leave your password in the tmux window, all right.

C

C

All right, okay, let me get this other other computer up here and I will share on the screen.

E

B

C

All right, I think I have just barely a little bit more disk space on this machine. um Yeah. Sorry great so.

C

Let's see how many more passwords I can display here in one meeting, okay check out confidence.

C

All right and what can we look at a specific test? That's failing.

A

uh The one with the sources.

A

Okay, I fixed the other ones. I just have to push it.

C

Where was our source test.

C

C

Here the confidence- okay yeah, the csv source- is exactly the one that I was thinking of yesterday: okay yeah, so I'm going through the csv source right now and I'm um the issues that were brought up. um You guys probably saw them, but but there was issues brought up around um um what is it called?

C

um Oh yeah, that guy alex he said um he's said something about: um um oh yeah, there's a bug, there's a bug in in the way that we were doing the csv source um and it made me think about how we have this. Let me I'll just open it and try um all right. So all right, so these guys csv headers.

C

So basically, this is the issue you're running into is that um in the csv file, the way we were storing the the the confidence we were storing the prediction and the confidence by prefacing the values with these prediction underscore or the prefacing the column names or you know the names that they were stored in that predictions dictionaries, prediction underscore and confidence underscore.

C

So what you're going to need to do here is you probably need to see like, and this is the loading so basically we're saying: okay, if you have volta prediction and confidence or something like that yeah, I think yeah. So if you have the prediction, you have the confidence, then you know set the value and the confidence within that dictionary right. That is the predictions dictionary.

C

um We don't want to do that anymore, though uh we basically, I think, we're gonna just throw, I would say, throw out the confidence for now um and we can evaluate what to do with that later. um Just set the prediction um and let's see where's prediction field names, yeah, so file, write out items get all feature names yeah, so you're just gonna need to work with the csv source here and I would say, just throw out confidence for now um and we can decide how we want to deal with that later.

C

uh I think we may want to add like we may want to explicitly have the csv force source say we will probably modify the csv source later to store confidence confidences explicitly otherwise we'll store the prediction. Okay. Now, I'm realizing that I'm realizing now that if we we, if we store predictions, we don't know which is prediction which is a feature anymore. um We should probably add- and this is this is- and this is how we would do it for confidence too.

C

We probably need to add something to the config here to say um you know these: are your prediction values right? So uh something something like you know. Predictions is a you know, list of strings.

C

And uh you know it needs to default to none, um because you can't default to an empty list. um Oh well, you can default an empty list. Yeah you can do field and then default.

C

Factory equals slamba.

C

Yeah, okay, so um this is this is what you'd need to do right? You set predictions, and you say: okay, the columns to be treated as predicted, values.

C

Rather than features and then you do the same thing for confidence right.

C

To treat it as confidence values does that make sense.

A

C

um Yeah, so you'll just need to sort of sort of massage that until it works um all right, great.

C

C

Great okay, so uh where did oh there's the nuts uh running? It's funny yeah? I just ran into that the other day I was like, oh god, jim's gonna have to do with this and uh well you know the easy fix, easy enough fix, so we'll be fine um and into issue with prediction: csv source, storing predictions and confidences.

C

B

C

uh Fix is to add uh properties.

E

Csv source to explicitly.

C

Set um column names names to be treated as predictions and confidences when reading all right, um all right great, so anything else in your end, section or uh hashem.

A

uh No thank you that was it uh I'll, try it out and uh get back to you all.

C

Right great all right so and then let's take a look at this data flow stuff. So um so.

F

660., uh yes, in that, like I, was actually trying to find.

C

F

To work with data flow, so I think, like this is the best place to start.

C

This is a meta issue: yeah, okay, so.

F

Yes, so I was thinking like uh like I was. I actually read a lot about data flows, but I'm still like not very confident like how I should.

C

Yeah, yeah, okay, totally totally understandable, um okay, so, let's see so yeah the idea. Here we have a lot of test files right now for dataflows. We don't have a lot of examples, um and so that's sort of you know where, where this is coming in um and- and I think that you know mainly- what we want to do here is is add a bit of a thing under you know, tutorials, and then data flows right and just build out this. um This area here right um and I think that that yeah so.

B

C

A different screen, thank you. Thank you, wrong. Computer.

B

B

C

Great so this this this tutorial section here under data flows, um and so, for example- let's see this is the I o stuff. Well, I think this is actually. This is actually this tutorial here. um So this one is, you know, oh, this is this one is good this one's already done so, but um okay yeah. So let's see um um yeah this one. Didn't we use this this this one is also sort of covered.

C

um The main thing that I think we need to do here with regards to data flow is cover the different syntaxes that are available um and uh but to sort of just you know, provide small examples around each of them, and I think this is a good. So this is a good example of how you would do one of these syntaxes.

C

um This ends up getting used in what was this? The flower demo right.

C

Suck champ this gets used here, yeah.

C

C

This definition no get single spec.

C

Yeah- and I don't know- I think we would be good to do some more python demos of this- um of the data flow stuff too. So, let's see.

G

Python demos will be easier.

C

G

To c11s and they'll be faster to make yeah yeah.

C

And so so yeah, because you can probably take a lot of this stuff- is already in the tests too. So yeah a good way to approach this, I think, would be to go through look at the tests that exist, um that that instantiate data flows and sort of pull out the content from the tests into an example file and then from that example, file run that you know, using the run, that example file using that using the console test stuff, um and so actually, let's see yeah yeah.

C

Let's do that um and I think you know one of the things that is going to be helpful here is um okay, so I recently added this um so yeah, we'll do literal includes and stuff, and then, where did this go? Okay, so usually the the um the okay? I think I need to document this too.

C

Where did that go test? Talk strings now, tutorials! No! This isn't test docs test console test all right. So this is the console testing stuff.

C

So this is a list of files that don't need um a virtual environment, so we can just run them sort of very quickly, uh because so the the way that most of these tests works is they assume that you know uk you went and you installed a fresh virtual environment, which I think we mentioned that we ended up mentioning and mentioning that in the docs I think tutorials says you know: yeah tutorials are best approached by creating a new virtual environment, um so most of all the tutorials assume that you've done this every time um so, but that takes time, because this goes and runs virtual first of all, creating a virtual environment takes time installing and updating pip setup tools and wheels and installing dfml takes time.

C

So uh instead, if you want to sort of do short, a shortcut on this and have a faster test and build and test cycle, you add it to this no setup here, um and this is, you know, list of list of paths that don't need a uh don't need that uh sort of extra extra stuff and, for example, this guy. um This is this double context: entry stuff that we covered. So this doesn't need any extra setup right.

C

This is all standard stuff from the standard standard um or the base package right, um so that that is how you're gonna accelerate your your build and test cycle there and, let's see yeah so and then beyond that you know, um git grep is going to be your friend here. um Kit grep is always your friend. This is great.

C

Okay, grip is great, um so, let's see test integration, operation data flow, uh df create where else is good to look um yeah and the other problem you're gonna run into here is: is you're gonna, see stuff from all over, um like all over our history of various wrappers around you know what we had as data flows- um okay, there's some stuff that I had in mind that I really wanted to see um recently. I'm trying to think of it.

C

So, okay, searching for data flow in this project is a bad idea.

F

I actually searched for data flows in the document. Oh.

C

F

Yes, because, like I, was trying to read up on some documentation.

C

Yeah you just get a million results, um let's see so where is you know there? Is this one that actually I'd really like to see this done, um and this is a pretty good data flow demo, um and I was almost done with this um so where'd it go.

F

There was an ice cream sales demo. I guess the.

C

Ice cream sales demo is is, could be a good one that one's actually that yeah. That could be a really good place to start with this, um so at the ice cream sales demo. I believe we covered this in a meeting at one point. Where is this? I think we have a recording of this um because, um uh oh what's his name.

F

Yes, we have like a long explanation.

C

Yeah: okay, great.

F

C

Is that's a perfect one to start with yeah, because I think, um uh let's see uh naeem yeah yeah, he he never got he. He never got the chance to do this, um although he wanted to- and I don't know if I don't think he's gonna get the chance at this point I haven't heard from him. I had last I heard from him he's you know: he's got stuff going on, which is great, but I don't know if he's gonna get the chance to do this demo.

C

um So and since we have plenty of explanation on that, I think it's a pretty good introductory one with data flows, uh because it's basically add this one operation and then you know, run the data flow. This is a perfect one, um so so yeah, why don't you sort of target that one and then let's go from there.

F

Sure I will try to work on that.

C

Awesome great cool um and then I will review that that pr sweet cool well um anything else from anyone yeah. Thank you.

D

Yeah, uh I want to ask what changes need to do in a light gbm pr.

C

I think we just need to update master on on those we need to to merge in master.

B

I have done this part. Okay,.

C

Great, um let's see.

D

Let's make a review.

C

Yeah, let's see.

B

C

It okay great, um oh today, what happened here? uh Something's failing.

C

C

Oh, oh, oh yeah! Oh! This is because of the h2o stuff that I pulled out of there all right. Let's see, let's see um yeah, let's, let's we'll we'll figure this out offline, I think we can probably wrap the meeting now um cool great um thanks. Everyone and I'll I'll talk to you on on getter and I'll talk to you next week have a going bye.

H