DFFML Weekly Sync, 9 Jun 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Weekly Sync: 2020-06-09

Description

Meeting Minutes: https://docs.google.com/document/d/16u9Tev3O0CcUDe2nfikHmrO3Xnd4ASJ45myFgQLpvzM/edit#heading=h.527dc3831vg0

A

If I accidentally left the recording, suction I suck show ya section on when we I started, recording a little bit of our one on one I was gonna, send you that stuff that we talked about and then I left it recording and I picked up. My laptop, like I think it was yesterday night and I opened it and I realized.

A

The recording software were so open and I had the buttons that stop recording instead of stearic or and I was like God know what happened and I think I have like a gigabyte recording of like I'm, not sure what cuz my laptop was closed. Most of the time yeah.

B

I was wondering if you closed, it all started: work yeah, yeah, I'll,.

A

Have to go through and I'll have to go through, it fix it see and then I'll send it to you.

C

A

D

A

Barely it's kind of kind of thing.

A

C

Okay, all right.

A

You stay the night and this is not Tuesday Friday. First, this is Friday. The first.

D

A

Don't know what do you guys think? Is that better or is it oh.

D

A

There you go now you're better, that's what you should sound like yeah yeah, that's what you usually sound like all right. Okay, so I think we are almost where we need to be with this stuff. Let's see, let's go look at the poor Chris, because I know we saw some movement on that recently.

A

Okay, so this is passing now Sudarsan I was saying this one and you're also doing this one right. Okay, so we'll leave this one. I will leave this one off since she's, not gonna. Call you, okay,.

A

All right nice work with the to commit titles. Let's see good does last thing: I, don't want to see ya.

A

Blow them it's first input section. We should really flip around parse input section. Let me get issue for that. I, don't know. What do you guys feel like I feel like it's kind of backwards.

C

A

I, don't think when.

E

I knew that I.

A

Meant to do that.

A

Okay and then yeah, so we did the quoted thing. So how does this end up? Working with in practice, yeah.

D

It works like I use, didn't that was the entry point or D okay.

A

Let's see, let's check that out: oh yeah and yeah she's almost done with all the airs on windows. That's gonna be great.

F

D

Also I'm getting this weird letter, I'll just end it again, you know and forward it it alright.

A

So the flux cat was just emerged. This.

F

F

A

A

A

What if the operation name has dots in it here, I think you wanna max what if three and.

A

Yeah, do you see what I'm saying yeah.

E

A

So yeah, so you need to handle the case where write this down on the nuts again.

A

So yeah we need to handle the case where K is something else here. So it's.

F

The Bell type equals inputs.

A

Yeah, you can split it, you could like split it. You could split it. Take the last two and then join the join.

D

A

D

A

Possible solution is to split use last two. Well, okay, so the thing is it's not always going to be the case. Yeah yeah, that's always gonna be the case, because flow is always gonna, be one level deep.

A

A

Yeah, okay: let's do these last.

A

Okay, I think that's it on this.

A

Okay, the one thing we also do need to do is out in this poll request, let's also update, so it's also update the.

A

Places where we use this so because it should be a quick change want to make sure we catch something.

A

Wow I love the search, not let's.

F

F

D

I want a higher resolution monitor seems really small. What like your big screen.

A

Let's see well I just changed that too I.

A

Changed it to be 1080p instead of 4k. Let me check the recording is correct. The other recording is correct, I don't know. Maybe let's see, let me maybe try stop presenting and representing, because I think I changed the resolution while I was presenting, so it might be a bug with that. Let's see good catch.

A

Is that better.

D

I'm not really not.

A

Really? hmm Okay: here you like.

D

It something yeah.

A

It's definitely my.

D

Monitor so I'll just.

A

Switch to this monitor it is this better.

D

Yeah is it okay.

A

Weird, it must not have skilled. There must be some kind of bug with Chrome's presentation, not scaling displayed correctly. Okay, once again, I gotta blow my nose, I'm, sorry guys something having allergies. You know.

A

All right, okay, all right, I'm glad we figured that out this guy's still good great. hmm This is all weird.

A

All right now, I hope today is not another. Just like technical difficulty day all day. Okay, get grip. Data flow, create.

A

Okay, I don't want to give some context, turn alright, okay, so that one's fine, okay, that one will need it Doxey a lie and then that one doesn't need it.

A

Actually, I think this is the one that suit honcho walled up doing how's it going ash. So let's make it note in Sudan shoes PR that we are ready for this oops.

F

A

So we'll be merging soon as soon as you make that change, then, once it's merged, we can update the create command here. Let's actually put that where that needs to be updated.

A

Oops usage data flows all right.

F

Where to go here, we go.

A

Okay, so in that case, we just need to update doc CLI.

A

I'll just make a list.

A

A

Examples, ms create dataflow to SH yeah.

A

Okay and then I think we'll be ready to go on this.

A

I'm sorry guys I'm having really bad outages. I gotta find something blow. My ass here I'll be right back.

A

Alright, ok, sorry about that! Ok, so, and then all right, so we're good on this just need these couple things that should be quick. So you don't have your ear feeling like that one's good to go right.

A

Yes, all right.

D

A

So let's take a look here.

A

Ok, so once we merge this and it'll only show the difference.

A

A

Solution, nice.

A

Yeah, we really need to figure out how to we really need to do that that one change where we're gonna, you know make the we really need to do that change where we allow config files to be used for command line stuff. Let's see, oh you updated. They used to look great.

A

F

Sweet sweet, sweet.

F

All right, sweet.

F

A

All right and then.

A

A

Here, all oh, no, maybe they don't! Oh yeah, okay, something died! That's right! Yeah! That's right! Just something! Non-Deterministic! You didn't change anything in there. So, let's see supposed to put operations, convert to give the output step result, dot.

F

A

Okay, yes looks good.

F

Put mode operation stop in order to gift I input style.

A

Okay, we need to probably automate this tutorial as well as suit how'd. You found out, we've got you know. These tutorials are kind of a pain to test, so we need to automate them at some point: check secret match, yeah.

D

A

Okay, seed secret up any equals any file.

D

These responses.

A

Let's see here.

A

Let me just one second here.

A

Okay, that changed nice.

A

A

Hh-Hey, this is not white I think we should do this. So I concern with the secret file name coming from seed right now is that the inputs also come from seed. The inputs from the.

A

Two inputs from the.

A

Give the inputs from the server well I, don't think it's a problem right now. We don't want to change them.

D

A

Well, though, I mean they can see, they'd see you'd see that the seed The Seeker file name comes from the seed right, but the thing is that this isn't supposed to be like the secret file. Name really isn't supposed to be something that is in input. It's more of it's more.

D

E

A

More of a config type thing right, so seed is something that might be an input at some point, but this is never gonna, be something that it that would be an input. This is always well so, okay, so the other thing is that.

A

The other thing is that the way that this was supposed to work I was realizing was one second.

A

So the way that this was supposed to work was, let me just sort of demo, so he ran or let's say we ran the data flow.

A

The idea was that you could populate the config of an operation using secret, the secret storage, so basically the configuration could be a secret rather than stored. You know in that in data flow file right under that config dictionary, so you would sort of just go and populate it itself, let's see so how should we do this I'm, realizing that we only did sort of part of this?

A

That was my fault that didn't communicate. That well enough.

A

Let's see so the best way that we could provide is is it's. When we run the data flow we need to be able to.

A

When we run the data flow, we need to be able to have secret storage in there and specify what things so just a secret secret store. It has to be with him like the running of the command.

C

A

I think for now maybe we should just.

A

Yeah, for now we should probably just put it in the config, but we need probably a better way to do this, I'm just so yeah. My concern here is really just that.

A

Number one you know things that are going to decide the values that we're checking things against should never should never be in the inputs I'm just in case they get mixed up right now, because it an input could always come potentially from a user, so we want to put it in the config, so the config was probably look like something.

A

Let's see. Let me just want to find this a bit.

F

Here so let's look at.

A

What's this branch name at deploy secret.

A

A

F

F

Operations deploy.

A

All right, so we have that config class reprise, just name it big butt and.

A

Oh yeah, the database stuff uses it so the same way that we did with the database. Where we got the was it there yeah we were getting like the the file name itself. What was it? Oh.

A

It's a cluster, this query config, oh yeah, that actually set the database that we were using app. So in this case, we'd want to config class to be like the secret right, so Chuck secret, just going to fake.

F

F

A

And then right it would already be instantiated, and so we would just say you know solve doc. Config soft I could figure was so if that config, oh.

F

A

Yeah and then you'd have to do imp enter yeah same stuff. Okay, that's right!.

A

So the same way, we did this context enter, do this sort of parent so- and these are obviously so- these were shorthand for when, because the operation is one of those double context, entry sort of type of things right, and so we do the double context. Entry on that. So that's.

A

So that we should just be able to get from SCT X, and that way it's the same thing that we sort of found out with, though with suction, was for requests recently, where we did the what was the oh yeah, the pre-processing, that this dramatically improves the performance of it too. So, let's see it's like you like you like all right so now in this case now we're always going to have it.

A

You know it'll come predefined, so we won't have to worry about the fact that you know someone could accidentally accept this the place to get the seeker from the user and then and and it'll, be it'll.

F

Be configured with the rest of davelis, let's see.

A

Like if we need to figure out how to automate this at some point too, so it configures probably not imported.

F

They're, just not sure about that facing port.

A

I will, let's see, let's also look at that data flow.

A

Let's see so this is the webhook data flow and you got that by generating it with. Let's see, this is the way that you generated it, and you know what we're realizing now is that we need the same thing as we have to the flow command. We need that for the config command.

A

So, let's see so. Basically, though, since you have once you do, let's see I think what you'll do for the flow command will basically just be well, let's see for this guy, it was it's Henry.

A

Okay, well, actually, let me also look at this, so this split dot separated here is split on dots and then you look for then you're looking for so you might want to actually do split dot, separated here too in case the last one is something that has its you see what I'm saying so in case. The last thing has yes, it lets you, but.

A

Yeah I mean you could pass it. Well, isn't it is this the single quotes or double quotes, or is this just single quotes.

B

A

Okay, okay, yeah, so lets you split dot, separated.

A

And let's see I mean well you're at it, since you have to modify this one, also throwing the config stuff and then so, let's see so because that should just be a copy-paste of like this to actually admit shouldn't be a copy-paste. It should just be for K, V and self that flow. You do split yeah, you do split, and then you just do the Traverse set on dataflow da config. So let's add another.

A

Fort for setting fig.

F

It should just be a TV in South. I can fit.

A

A

Split up separated.

C

Oops, okay and then.

A

A

All right that should be that I all know, don't add a file all right and then what you can do is you could set the any foil any file here right by doing config equals what I said it will be the creating a secret dot where's. What.

A

Oh other data file yeah. It's this one check secret match.

F

A

Let's check a secret match that secret dot.

A

A

C

A

And this Traverse set create dictionaries if they don't exist, just.

A

Not so current level equals little current.

A

Let's see so, we should probably change this. Okay. I know this isn't in the right one. No, it is okay, because we're yeah, you're gonna have to you're. Probably gonna want to pull out the commits that you used in this one or you're. Gonna want to pull out you're gonna want to pull them out of here like use, rebase and delete them, and then use rebase and re-add them once you merge once this is correct right. Does that make sense? Did you do them like that? Okay, cool, because you're probably gonna, want.

F

A

Right, okay, great so, let's see and then you're probably gonna want to do right. Something like if not level and current.

F

A

Or well, it's just what we're saying here.

A

We're a little yeah because we want.

D

A

D

A

Just saying yeah so I'm saying here, what we're gonna want to do is that secret config, dot, file, name right- is this so, and this is any right so because what we're gonna want to generate is big.

A

A

A

Plug-In is any rate and then oops.

A

A

Oh man, I, wouldn't that's.

F

A

All right, I'm, sorry, let's see, let's wrap this up. I think that's the last bit of stuff on these pull requests. I think this sort of gets us to a place where we've got everything that we needed to talk about here is now yeah.

A

Now it's in the config, which is what we wanted, okay, so and we may actually want to be able to do things other than that later I'm like specified when we start whatever command, is gonna, be running those data flows but we'll cross that bridge when we get there, we definitely want it to be in the config for now so.

A

This might not be correct, but we need the point is we need to be able to create dicks if they don't exist on the way down.

F

A

That's alright cool right, because you're gonna do config top filename and config won't exist, of course. So, oh and actually yeah, you know those weren't exist either. So, okay, let's see here, I, think that's it alright, I'm, sorry that took so long.

A

So who else? What else do we got? I know we got a bunch of people on the call who wants to talk about something.

A

Or augen does that sort of settle all your stuff? For now.

D

A

D

A

Right awesome all right, thank you. Oh.

F

Use secret from config.

A

Okay, so who's who's up next year. Sacrum did you have any updates on anything.

B

Yeah, so I was trying to train them. Nice dataset, using psychic models and in psychic in psychic models, dot, PI I found that models only support single values. There's an error for that. So psychic doesn't support rare.

A

B

A

B

A

You make a part for that.

B

Also so I commend it that out and and then it went to psychic base, and there was some stuff happening with data frames and everything and it was not, it was giving me an error.

B

I can share my screen if you yeah.

C

Let's check it out.

C

John should.

B

We get rid of the applicable features and cycle.

A

Yeah we should give very, very feature stuff, yeah, I, think that was that's an open issue that we have is we're trying to go through and make sure we get rid of applicable features, because we realized that the reason why we had that was right, because when we before we had the dataflow stuff, it was like you had the features and they feed into the models. And sometimes you know you can generate all the features.

A

And so then you would end up without some features, but now well, obviously you want you want to generate all the features, so it doesn't quite work the same here. Let me stop resetting that.

D

Most because in there yeah.

B

Greater than supporting the value.

A

That would make sense.

A

Guys I do have a hard stop here at 10th. Unfortunately,.

F

C

A

Classic shaman might be worth running this through just for when you're testing this stuff. It might be for running it through the Edit command right, so you can save it without using the data flow or I don't know. Maybe you already did that.

A

C

A

The normalization I'll get flow diagram.

C

A

Must be a string or number not topple and I take it. The tupple is that array.

A

C

A

Is the giant array, okay,.

C

B

I was able to fix this.

A

B

So I don't know what hello yeah.

C

B

So yeah so I was I just found so that so so that the MS dataset was at least being trained. So the thing is that when the when the tuples come so and I commented this out, so that so that this error won't show up again and again. Okay, so here like when the when the X data comes, that's that up alright, so it forms a list stuff list list of the pool. Okay,.

A

Wait I'm not seeing any changes in your screen. Okay, there we go I.

B

Might be having connection issues I think yeah.

A

Maybe can you tell us where to look.

B

The inside Kate what in psychic base dot file? Can you see my screen? No.

A

A

You try representing there.

B

C

No yes, now this here.

B

So the X data is that upper right, so the so here the X data is a list of the list of double mean.

G

B

Should only be less because it won't be, it throws an error. Okay,.

A

B

A

A

So you commented out, you commented out that stuff.

B

Yes, so it's when I do this: it.

C

A

Psychic context: okay, I! Don't have that wait a minute, so I.

A

Mean the stuff that you commented out does not exist in what I'm seeing here. Yes,.

B

Louie doesn't exist, it's like a puppet on the phone, because the edge is a list of tuples. So if.

F

You uncomment that then it works you're insane.

B

Yeah, that's, that is something I added. Okay,.

A

Okay, wait a minute! Okay! Let me see, let me see so xx. Data is a list, so you go through X data and you append the first.

A

You append the first in index of X to it. So what does X day look like before you train here for one record like so, if you were to train on one record and log X data thanks, I mean cuz. Right now, it looks like you.

C

A

B

So inside a list there's there's only one list and.

B

I'm saying that the X data here is a list of tuples, instead of only being a double.

A

B

Think it is a this some kind of like here in ID, x3 and ideas ones dot by ax.

A

Yeah, that's what I'm thinking here is. Your future name is with features now what you want.

A

B

A

Okay, so can you paste in the command to run so I can get the same error.

A

Actually so because we're gonna have to I'm gonna have to go soon. um Cuz I got a meeting rescheduled for right after this, unfortunately, I can't can't be late. um Why don't you create an issue for this and document so document document? What you're finding.

B

A

Commands are running and what changes.

A

You made that make it work.

F

A

I would say that you can do so. Let's see I'll present again and you can see what I'm about to do, which is I mean. This is the same thing that I've usually do right, but so for right. So for what we just talked about with OGG and he's gonna need how old got demon.

F

Wow, okay, it's not going.

A

Okay, so if you paste in the issue- and you can write in in the markdown formatting right this, so if you do diff and then you do get diff right yeah, so you did, we run git diff like this, and then you just paste all this into that diff section then it'll show up. You know like this, and anybody can copy paste this.

A

So, for example, if I wanted to sort of start working on this issue that you were that you were talking about, I would have the commands that you done, and you know the commands that you were right typing into the terminal to cause the issue right and then so I'll just reset this, and so now I don't have those changes right and if I wanted to grab those changes that you had I can copy what you have there and you get apply and then I can just paste it in and now, if I do get DIF know, I have those changes right so that'll, let other people.

A

You know if that, if you, if you, if you do get diff and copy paste that stuff that you commented in there or wolf this stuff, that you uncomment it right when it's working, then people can sort of try to debug or you can try to debug. If you pick it up later, and you forget about it right.

B

A

B

A

Thanks and then, if yeah, okay and then so it's is, anybody else have anything we can jump on real, quick, so sweet, Honshu, I think I still haven't gotten the chance to go and look at what's going on because it looks like you were seeing some weird issues there.

A

G

Like there was this missing definition context.

A

Okay, yeah so also take a look at that. Do you have any ideas or.

G

So like I had this question like like: is it expected at all? It shouldn't be there well.

A

G

Definition, nautical context.

A

Safety check number of issues is basically saying that we ran the data flow and it tried to look up yeah cuz, we're in the outfit operation here. So the output operation went and tried to look up safety check number of issues, and it didn't find it. So. Oh, it's quite possible that we actually maybe should have updated the let's say, chick safety check number of issues. Is that actually still what the you may not still have been, what the.

A

What the definition name is.

F

Okay, let's find out oops, oh.

A

Yeah, so it should be safety checked out outputs that results. That's why so in our seed.

A

Yeah, so in our seed we actually need to change this to safety, check, outputs that results, banded output becomes run, bandit, dot, outputs. The result, that's what's going on here and then language to comment ratio might still be the same.

A

Yeah language to comment ratio is, it looks like it actually disappeared from this one yeah, okay, but I think that was because then it goes because we we didn't have to flow working when maybe you regenerated the output here, yeah, okay, so I think that's what's going on there. Yes, yeah all right sweet, so we figured that out. Yeah.

A

Is there anything else? We were so I think you're just waiting on the flow stuff and Hagen. It sounds like I mean that those changes there were. Basically it sounds like it's pretty minimal and the flow thing, and then the ffmpeg might be a little bit heavier, but.

D

A

Great I figured yeah, I figured it'd, be close, all right and then so Nayeem is there anything you wanted. To recap. I think we had, let's see.

A

We needed to rename.

A

Definitions in.

F

For back or spec output, operation.

A

And seed will update, create commands once it is finished, PR, okay and then, let's see so naive, Hey.

E

A

I said: oh no, that's right! Yeah.

E

I did if you could tell me briefly how many data samples we should change it for okay,.

A

E

A

Mean so I think what you're gonna want to do is you're gonna want to probably you know, you're gonna, wanna I think, let's say I think I would grab like you're gonna automate the creation of the data right. So it's sort of gonna, be it shouldn't, be more of a task to grab more data than it would be otherwise right so you're gonna, remember: you're gonna make that linear function. That basically says okay.

A

If, if my average temperature is about this I'm going to you know I'm going to generate you know if my average temperature is X, I'm going to generate ice cream sales corresponding to X right and then we're you're going to write. So that's going to be sort of like that's your your random data generator is that linear function rate and then you're gonna find you're.

A

Gonna want to find so you're just going to want to pick cities at random and you can probably go to Wikipedia and scrape a list of city names right, and you can just then go and feed that, through your weather data grabber for each month and say, okay, what is the temperature? Now you run that through your fake data generator, you know your your fake linear function to generate what the ice cream sale is and then so the reverse of this right. So the demo is actually to go predict for other cities.

A

What would the linear function? You know predict the sales made it to be right and then, if somebody wanted to, they could put in their actual sales data. For you know, XYZ product, so I would say. Basically the main concern here is figuring out how to grab that temperature data and then- and then you you just you know, make your little fake, fake function and you just feed a bunch of cities through it and because you're gonna be running this operation, that grabs temperature data for city and month over and over again. So it won't.

A

It won't sort of be any more hassle to grab more records. If you want to right. If you want to create more test data, you can just sort of play with it. The results are all going to be sort of.

A

You know arbitrary right, okay, yeah so, and- and it should be- you mean you should be getting good accuracy right, because you should you know if you run this through something like I, don't know, you're gonna want to run it, maybe through some of the psychic models, writing and and and find out which one produces some good results, or do a few of them where yeah just just do one but yeah I have to run now, but okay yeah and it just ping me.

A

If there's any other questions on this so ash did you have anything you wanted to update.

D

A

Specify what config loader we're using oh yeah, yeah yeah! Oh.

D

A

Well, yeah that should probably be changed to config later then so yeah. We should probably change that to config loader um I guess that's not gonna be a that's kind of annoying to go through and change all.

B

A

But I mean that's basically only every time there's an invocation of data flow create with config, which there aren't going to be tagging.

D

D

And I'm willing to die yeah.

A

We're gonna have to probably do that eventually right, so cool all right. Thanks guys, sorry I have to have a stop today here, but all right, I'll see you guys online and ping me if there's anything else. So all right thanks but lay thank.

E