DFFML Weekly Sync, 22 May 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Weekly Sync 2020-05-22

Description

Meeting Minutes: https://docs.google.com/document/d/16u9Tev3O0CcUDe2nfikHmrO3Xnd4ASJ45myFgQLpvzM/edit#heading=h.9mh4wtq03h8v

A

Okay, sweet recording.

B

C

B

So I think what we're gonna have to do is probably just go through and- and um you know I guess the problem is- we have to include log and everything god damn it um so.

D

Yeah, we have to include log everywhere.

B

That was part of like the fun is. You didn't have to include log, uh um maybe.

D

C

D

A workaround for a specifically log.

B

Yeah, you know, I think so actually now that we're thinking about it, um because what do we do, I mean a log is like a basically yeah, oh my god. Okay, we were making this or I was making this way harder than it needed. Be, I'm sorry big, surprise, that's always how it goes um so.

B

Yeah, it should just be command: uh util, cli, command,.

C

um Okay, yeah, it should just be basically. Where was it.

B

Yeah, okay, so subparser right. So we add a subparser.

B

Which, okay add subs, which one is add: subs? Okay, so, okay? Yeah? If this thing is another command class, we come in here and we say: oh, why is there no coverage on this.

C

If parses is none.

B

Of commands- oh yeah, this thing um so yeah we come in here. We basically we call add subs and subs ends up in here again and then we just add argument so basically at the first. Basically, this is all we needed to do. This is all we needed to do right here was just come in here and always add argument lock right I mean I'm thinking.

A

B

Right, okay, we're making this harder than it needed to be, uh after all, like log, is hidden from everything, so it really should just be all contained in this file um sweet. Well. This is great news. um Let me just make a little comment and push this up. So, uh let's see self yeah, so in order to add log to every command, uh we just okay. Let me just add it here.

B

First thing I mean that'll do right so.

B

Wow, I'm glad we figured that out. um I'm sorry, I'm I'm stubborn on things sometimes sometimes sometimes uh we need to. Let me know if I'm being ridiculously stubborn, for no reason on some some things, because.

D

No, I wanted to work it like that too.

B

Yeah right, it was like it felt like that would be great right.

D

It would just keep the things the same as they were.

B

And it would all look so clean, oh well, uh okay, so to do log, um and then let me just push this up here and let me make a note that, can you take this from here then basically go through and and do the rest of them.

D

Yeah yeah yeah, okay,.

B

D

The thing is that only log will be added and I'll have to add uh things like. If there is sources arc source is being edited, then I'll have to just add it again is somewhere else where this is.

B

Being edited yeah we'll.

D

Just have to copy paste, it.

B

And I mean one thing that we could do is we could just like define a bunch of fields right, um because we have the field um thing and so basically, like you know, equals field.

C

Let me just say what I'm talking about here.

B

Right so, if we're doing like equals field, config loader, we could just say um you know like we could have something in util cli. That's like a bunch of just um fields that are predefined, so you can be like default blank right or like blank field.

B

So, for example, if we had.

C

C

Let's see what's a good example of this um yeah, I guess sources would be the best example. um Yeah sources is the best example, because so many things use it so.

C

All right, so let me just do you see what I'm saying here, or I guess I'll just make it up on it like clear just in case.

D

I I don't and I'm not understanding what you mean.

B

Right now, okay, great so I'll, just great as in I'll finish my example. I wasn't wasting my time um so, let's see so.

C

I guess we should probably call it field sources.

B

So we can do something like this right, where, if we want to reuse, we can just do that and then import them from so this would live in, would live.

C

D

Okay, so the arguments that are most used, we can just make them our default argument, type. Something.

B

Yeah exactly right so, and we probably would want to put it somewhere like here right, because no this is config stuff rather than cli stuff. So in that case, right yeah, we can just say field sources, and you know that might reduce the amount of repetitiveness that we're doing. I mean we probably should do that where applicable, um because if we ever have to change something, then we don't have to change it. Every.

D

B

File right um so anytime, if you're running across things and you're noticing that they're the same then probably just stick them in into into here and uh and then we'll be. We won't have to change them later if we have to change them. For some reason. Oh.

D

C

um Okay, so let me just uh add that uh split to do split out fields.

C

B

I added this. Actually, this is something I need to do is oh wow. That screen is really big.

C

um Okay, I'm really used to having oh it's zoomed in um I'm really used to the 4k. On this monitor, it's really throwing me off. um Where did that go okay? This is something that we need to add, and this might be helpful for you guys where the is here there. It is.

B

There right so this right here um I'll make it a little bigger.

B

I love linux machine. No, I just locked my screen just as saying how much I love linux, um because everything just works. um So this basically uh do you guys know what commit hooks are and get or like hooks and get uh no okay. So a hook is just check out the git directory sometime and in your ample free time right.

B

um So the git hooks, if you look in any git repository in dot, get slash hooks, um you'll, see a bunch of these things and what these are are just little scripts that will run as they say before, whatever action they're talking about right. So if you say uh actually, I haven't even what is this okay? Well, I have limited experience with these, but whatever this one is looks cool, um uh let's see so.

D

E

Things like that before that.

B

Yes, exactly right, so if you say like cat dot, get or vim dot get hooks pre-commit.

B

So what we've have is basically just exact, execute black and and check right.

B

So basically, because we're doing exact it's going to fail, if, uh if, if you, if like, if, if so because we use exec, it's going to replace this bash script with black, whereas if we didn't use exec and we called black, um the return code of black wouldn't be checked um and the script wouldn't fail necessarily so it wouldn't stop git from from uh uh not committing unless we do set e, which means like turn on every time, if, if a sub process that you call from black or from bash um exits with a non-zero return code, so because zero is success as far as um unix posix stuff is concerned, like processes returning, so if it exits non-zero, then the whole script is going to exit.

B

um This is another trick is x, is going to echo everything that you do so, if you ever want to know what a bash script is doing, that you're writing put set x at the top and if you ever want to catch it as soon like writing bash scripts. This is very helpful because you do set e set x and now anytime, you're gonna see every single command being output to the screen, so it's gonna print it out with a plus in front of it, and it's going to say you know whatever.

B

So that's very helpful for debugging and then also setting error is helpful because you'll know like because some commands don't output, anything if they fail right like if you do grep for something and it doesn't find something, and you told it to pipe it to a file. Well like you're, just gonna have an empty file right, whereas if you did set e it'll exit because um it'll be like, I didn't find anything instead, like the next command that was going to do something with that file.

B

Full of stuff is not going to run and because the script will have exited once bash or once grep didn't find anything to pipe that file, um so set e set x is helpful, um but otherwise, if you just do exact, then it's just obviously going to replace it and therefore the return code of black will be the return code of the commit pre pre-commit hook and therefore, if there's any errors, it's not going to. Let you commit and you can reformat. So that's a good one.

B

I need to add it to the contributing, um but you can do all sorts of stuff. I have no idea what you can do. I just know pre-commit, um there's, probably a lot of other cool things that you can be doing here and while we're at it, we might as well just check out this one um watch. Man.

B

Well, this looks cool, um it must be something new and it's written pearl, weird, um okay, uh so yeah so you'll get that stuff. um Well now let me push um so such time. You've got that and then what else are you working on these days? Suck sean.

D

um I haven't, I haven't been working on the mnist stuff or because of my deadlines, but I'll get you.

B

It soon cool all right, that'll be great, um because obviously yeah that that I mean that wraps up that little thing very nicely. Now we can do the normalization.

B

I think the one thing was, let's see so when we're creating that we're basically going to be able to create the data flow with the create command, I'm not taking any notes, damn it.

B

Okay watch out sorry I'll be right back. I.

B

B

Oh firefox now securely routes. Your dns requests whenever possible to partner service to protect you, while you press all right for dns sec.

C

um Okay, so let's see.

B

Oh I'm not presenting.

C

Let's see, let's see, can you guys hear me.

D

Yes, we can hear you.

C

Cool sorry, I'm gonna mess with my video conferencing abilities today. um Let's see.

C

C

All right, okay,.

B

All right, oh no, it's super bad all right! Well, my other computer is frozen. So, let's see this is today is not a good day for my video conferencing. I'm sorry! um Let's see, I guess we don't really have that much to talk about. So basically, we've figured out how to get unblocked on the config stuff um we um you're going to do the mnist stuff, we'll normalize it. I think what I was going to say is that I wonder if I can share my screen on here- probably not.

B

No, of course not. Why would I think that would work, and now this is frozen.

B

All right, I swear my internet works. Sometimes um I just tried to share my screen on my other computer and I froze that one okay, so screen sharing and google meet is not a good plan today, um but basically, okay, so you're on we're unblocked on config stuff mnist will uh will do the data flow, pre-processing storage source um and um the.

B

So yeah, that's that's what I'm thinking is is we should do basically what what we should do here is we should oh.

B

Oh great, hey sue hunter how's. It.

B

Going um so what we should do here is uh we should use that create command, um that the modif, the one, the modifications that again just made to the create command to add the seed um we should have. We should use that multiply command or multiply operation that you already added um and we should pass as a seed input, the the multiplier and then have the uh let's see or do we need this stuff to modify the input flow?

B

I think we do because, or else we'd have to hand modify the input flow when we, I think we need to modify the create command first before we do the rest of this tutorial, um the rest of the modifications to this tutorial. I think we need to change the create command. Do we have an open issue for that?

B

I think we need to change the create command um so that you can modify the input flow on the fly like while you're, creating just like how you can add seed inputs, um and I think we might have some notes in this either in the issues or on. I think I think, there's an issue for this um or it's in the media. Minutes and sudanshi I'm having lots of issues sharing my screen today. So it's not happening, I'm sorry.

B

um So I think that needs to happens first, because we'll want to show the create command we'll want to we'll basically want to do the create command say that we want to get the input for the multiply operation from that feature and then we'll want to have we'll want to use that associate definition.

B

So we'll want to have the create command, be associate definition and multiply and then have the um have the.

B

Have the seed be the multiplier and the modification to the input flow on the command line? Basically, just saying that, okay for the multiplicand, uh the first argument of the multiply operation um grab that as that feature right, does that sound like a plan.

B

Yeah yeah, and I think we have some- I think we have either an issue for it with. I think we talked about using the parse inputs action, because that splits on an equal sign and then we'll maybe need to write like a converse trip or traverse set operation, which would be the inverse of the traverse git, which is already in data, um and that, basically, you can just butcher up uh traverse config get and you'll see how that works.

B

um But that's probably the first step um here and then, because that way we can show the creation all in one line, which means that when we, I think we have this in the unit test right and so in the unit test or the the test file in the examples directory for under mnist or examples mnist and then test file. The reason why it's important to get all this stuff into command line things rather than file modifications is because it's much easier to automate the tests right.

B

um So then we can just throw in that create command into the tests um and we'll we'll uh we'll make sure that way. It will always work and we have to do some kind of funky file. Editing right um within the test. uh Does that all sound good.

B

Oh yeah, let's.

A

See, let's see if I can open this without something exploding yeah. So, let's see.

B

um So sutancio how's it going with you.

E

B

So I yeah you were doing good stuff on these uh adding stuff to no.

A

F

F

F

F

The fact that people don't have to learn one more thing.

F

So that would be awesome and then I think oh yeah, I don't think we have an issue for this. Yet.

A

A

E

E

Wow all right, so this one.

F

Is this issue is to track, um oh when we load, so so? This is. The other thing is that um when we load the operations.

F

F

F

E

E

E

This is what I'm talking.

E

E

E

E

E

um Is tracking effective now that we have this entry point style, those stuff um um we can make it so that, like.

B

When we're running through examples and we're like you know, having.

F

People create operations for the purposes of data, close type of thing. You don't even have to write anything up on it right here. Well, they don't have to make it a package, because we've got these entry points downloaded and they can just say that it's a file. This is a function within.

F

They won't even have to add, opt to it. Basically, if they just do their type type, hinting stuff onto the function and stress if they have a picture.

F

Because, well we can do it at any point, but.

A

A

F

F

F

F

F

F

F

F

So this works perfectly for our web ui, because, obviously you know it's just a front end. That's those pages set up with people like who's gonna have big data and stuff, and now people will actually be able to. You know sit out there on the server paste in the urls, like you know, play with it for 10.

A

Minutes um so I'm going to try.

F

To get that situation working.

F

F

F

E

So these are the things.

F

F

F

Oh yeah, oh yeah. Well, so we have that's. That's all surprise promise. We have two things we're talking about with secrets right. We just talked about uh before this here and I just talked about the secrets: plugin, the secret plugin, which is going to be right for saving and loading secrets from from conferencing datables and things like that right and then, but right now, we're also having um just the support for the secret weapon right. So you need to generate a random value.

F

Put it in the github thing right if you're adding a web book, it's.

F

Okay, okay, yeah, so.

A

ah Well, I guess you gotta do.

A

F

I completely missed that one yeah, well, okay, so that secrets, uh plug-in type is gonna, be real helpful. um Let's see yeah, I guess it's the same use case too. It's just the conveying operation so um yeah I mean I don't imagine that's going to take you too long, though, because it's basically just you know it's adding a new plugin type, so you just need to do. You know make dirt. You have to come all secrets and then touch it.

F

F

F

uh Basically, it will take it'll take uh you're gonna. This is gonna, look very similar to.

A

Actually, what.

F

The keyring api looks like um uh which makes me think like do we just use?

F

Oh, I don't want to just use the current api, because the problem is that's the thing about the uml is that we have the double contest entry which lets you connect to and authenticate to any asynchronous services, which is nice.

F

D

F

You look in vf base and the other memory there's a key value store class there. This is pretty much the exact same thing, um so you can look at that for a reference.

E

B

um And as we get into this, I think one of the things we're.

F

Going to want to do here is.

E

F

And use that password to encrypt.

A

E

Just declare that this is a.

F

Dumb insecure secret source um because I will get a ton of crypto compliance class pass throughout me: okay, great um wow, good job, putting those two and two together there. um So all right, um I'm sorry! I said I wasn't gonna take a lot of your guys's time and I took a lot of years this time. So is there anything else we need to talk about.

F

All right: well, you guys know where to find me um I'm on twitter. So uh just let me know if there's anything, if not, I hope you guys have a great weekend. Then you can get some relaxation and.

F

A

uh The input stuff.

A

F

uh Yeah you're good, well you're, basically going to want to.

E

F

Training command so that the source that you're giving it is the data source.

E

F

E

Up the data source is.

F

Of course, that was there.

F

I feel like that. Sorry. That sentence probably sounded like nonsense.

B

D

F

So you're saying we need to flatten them out too.

E

F

Oh, I see what you're saying: yeah: okay, yeah yeah.

F

So, let's see let me look at the yeah so we'll we will need two data flows, then yeah that makes.

F

B

um Going to update the um I'm going to update the issue, this beautiful create modify with what I think the.

F

Mnist example is going to look like.

F

Issue with what I think, the new train, command's gonna, look like, um and that may or may not be correct, but how many people help me? People.

E

F

Oh, that is the confusion. Yes, okay, um can you just make that new issue then and I'll comment.

D

F

That issue with what I assume the drink I mean. Do you need me to comment with what I think the train command's gonna, be here or do.

F

Okay, all right that sounds great cool.

F

All right. Thanks.

F

Guys go well I'll talk to you next.

F

F

F

G

G

G

G

E

E

E

E

F

F

um Two different computers get failed in several different ways on each computer, learn how to do stuff.

A