ONNX Training Working Group, 17 Sep 2019

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: ONNX Training Working group meeting 17 Sep. 2019

Description

Recording of ONNX Training Working group meeting 17 Sep. 2019

A

Hello, everyone thank you for joining our own Ukrainian working group meeting today is September 17 and we're going to continue discussion or.

B

A

For sex training so a last time we're talking about raising, possibly creating more examples for our testing. A region will have any update on sir.

C

C

So, first of all, uh you know in our last meeting of I.

C

Merch is called a training reality, but one branch- the Megadeth as the protocol, so that everyone can have a single place to get everything where I discuss and that PR is. It is ready to be used.

C

That night has some tests failing because of the file format was something, but if you call into Chrome that branch you will be able to do this without the Linux and here's the full request. I sorry.

A

Do you want to share your screen and show something.

C

Yeah sure thank.

A

A

C

Okay, when I can everyone you agree. Yes,.

C

Oh here CPR, that includes every fear we would have told a lot. It includes all the optimizer, the change of the proper format, training, bakoto and the love song hero and aquarium gradient operator. If you want to have a clear view of the low signal, 15 acres, then.

A

C

The file, trenches and.

C

C

Operators guys b5.

C

I like building you so, for example, here is the organizer. We have bucket up great Adam.

C

And the momentum here is also great in creating gradient.

C

Let's see what this may look like.

C

So, basically, it's just it's very similar to other of our present definitions. But if you read this file with the providers in a clearer way to see the actual signature where I proposed, although as gradient gradient, gradient, query.

C

A

C

It is, readability is better than pure take well, if you want to review it.

C

B

C

I run right, we're going to put comments and merge these PRS into their local budget to see those signature are working with, while working for it for Ian I. Okay to think this PR, which the better branch and hope it'll be a centralized place for all the training cells, because the as we have discussed previously, it's hard to have the girl Bob you to all the training stuff in Marco Aldo separate his PR.

C

That is my prospect for the second one. It was good, so actually I feel previous I have sure something before Ryan letter. This is a journal open web for generating model, but.

C

There's a run, try this or graded before. Oh, it's, okay, so one comment got from fraud discussion. Is that it's hard to see what the actual training were so I I didn't so I further added function to create the model purely for training? No reason it will extract the training algorithm from the model and put it in another model so that you can can just visualize it so.

C

Yeah, here's our fan quest bilingual. We should start seeing you but- and it is.

C

Yep so again the four is stealing a function because green or is for defining I guess there are some universes right, limited hours. If you woke up, you will get out of the not really to the training itself. It may contain terms of node.

C

Okay, so if your difference function and you compute loss from your inference, output in the Big Show level and you'll get a lot better here in the grading operator, you compute the so the breeding of white label again the load varies.

C

And this outfit, though reading of the wedding and in the optimizer it gets, the is a way to be updated and the gradient that way. The gradient coming from gradient operation here is N squared here how to score linear way. We're getting out of scores annuity in the state of the secondary momentum, stay on top of the linear way and the something lighter come with new ways and new signal. The momentum from color input and you get the new one, so I hope it does.

C

It can see a more everyday friend anyway, plus two to see how the speck with work.

A

That's why the good sound, clear so much aeration, so that picture isn't included in some documental you just rollin.

B

Well, I will just confess.

C

The list by all right early on so fair, so every one hand tried it. It doesn't handle every case it because 240 generation graph view on it. Some of that is programming language tuning. You need a compiler, but it is at it is everyone an overview to how to use those newly introduced, the components to combo the model.

B

A

Simon had a question yeah.

B

D

Hell I think here mister additional funding it, but it was good why the lost value is an attribute.

B

What is which one or is not attribute.

D

On the gradient, I think.

C

On the gradient, so you mean.

A

All right question is: why is the lost well most value and attribute, rather than an input? Oh.

C

Okay, so let's grading operator use attribute to extract the function to be differentiated, so you can see here. What's the function we need to differentiate.

C

It actually start with to include include label and ended with the invoice, but history shows that inside anyway, let's start with three inputs: what's the actual commutation, it takes these three inputs, fit them into inference, functions and also sure and get those off value. Okay,.

C

So this actually means that we extract the software in the model and contact a psychopath of them.

C

So this is real criticism.

D

Sorry I think quite some of the last bit so you're it's an attribute so that it can work out what subgraph or what the inference function is. But.

C

The surra turns the big lines quite naughty naughty: okay, Ruby.

D

D

B

D

B

D

And so it's an attribute, so you've got lost value of an attribute of Y. That's the name of the temperature regime. Yeah.

C

It's an integral tensor and the actual value.

C

Would be computed from so there's anything I think I were healthy, so the motivation behind this is to separate the differentiation from below use different inputs in the derivative. Then you already know in good.

D

Sorry I'm done following that and.

C

Okay, so come on, you have a better clean, great location in mine. Well,.

E

I climb up the I'm wondering why I.

F

E

We discussed all this before, but the last value is not an input for gradient. If.

C

Not because it can be computed from other in good unable in a linear way, and we can. The data points see you later. Okay,.

E

um Anyway, yeah I mean he ignoring the exact details of the representation, so so I guess the gradient. Where to look at this.

E

ah It's computing.

E

The gradient of some variables with respect to some other variables and in fact then doing the back propagation and returning the output value right. That's the interpretation so.

C

So I guess special version is that why not only need to be an attribute, and then my answer to that is that we use attribute to define a software that we want to differentiate.

E

D

You work out what the sub graph is.

C

Yeah, you don't need to do the differential to the hood, where you can just find sub work of that and do the differentiation, because sometimes in your pipeline, some operators are not differentiable. For example, feature engineering cart, for example, careful. Yet they are not vegetables but after those three to engineering to finish your unit build neural network models with bit is original, but it's just a subgroup of the whole Python.

C

F

Okay, I can probably explain a bit here. So, oh, this is a journal from two enabling a representation of the gradient graph between AI for any sub graph. So like. Oh, the gradient doesn't always stop by propagating from the last function it can like. It can start in the middle, and then it came back propagate to a certain point where we need the credit. So the the attribute of this gradient node is just defining the starting point and ending point of those.

F

Differentiation.

D

Yes, I'm saying that it's just yeah this.

E

Direction so why.

D

Did the lost and you need to be actually provided, not an infant secret.

B

C

It it can be confused from the three hood I did it here. If you have input in your way, then label you can already computer all day. So.

E

C

E

Is the constant that is that is being passed here? No, no, so I guess the other way to look at this is there are two stages to this computation, so, first we need to construct the sub graph that computes the gradients of the BAP revolution, graphs, so think of this as a kind of a higher order function. So you specify the Y and X s and we construct a sub graph that corresponds to the back propagation sub graph. And then you need to evaluate this back propagation sub graph for specific inputs and outputs.

E

So the model here is the attributes are used to specify the.

E

Info required to construct the bab propagation sub graph and inputs/outputs are used to supply the actual inputs to actually evaluate the baton.

D

Refresh and sub.

E

A

D

A

Little maybe a strange way why the gradient is not connected to the inference function. Oh.

C

It's increasingly connected to here, you can see the starting point is linear within the label and the any point slope and airy, and the three Sonny's the one two three, the three variable here. They form the code scope in the your model and we'll just take everything. Let's go pal as the differentiate as.

E

The sole source.

B

E

This is okay, but the graph you extract, your your script generates right from there original model, yeah and so I. Don't know how this relates to the other one. Whether some of these things have changed. So the gradient node looks identical in both cases.

E

C

Does it mean it's identical to which case.

E

E

I mean I mentioned: you had written a script which will take the graph proto that we have designed and it it transforms it to produce this for viewing in a plan right, you said yeah.

C

E

Just wondering.

E

Investor is the site, the gradient note that appears in the original graph fathers yeah.

C

Okay, let's in my script, my super does direct addressing first, it creates a training model along just to fit the training model and second, that is extra part of the training model at the core of us, email or character.

C

To the Peoria question why it does not connect you to the inference square. Well, it's increasingly connected to the empress' coil inference squares, because here the the graph we are going to differentiate starting from linear way the label and ends at dots value, and you can see Starbound include label. We basically have been. We can touch all the inference, function, log functions and funny and at the lost value, so we can make increasingly connected to them.

E

So I guess one way to look at think about it is that that black box called gradient includes a copy of the left-hand side right yeah.

D

So yeah, so we should think of a bit. It just seems like with duplicating information, because we have the names of the input tenses as being inputs. Why do we need the names of them duplicated in the attributes?

D

C

So this for allowed for allowing different inputs, for example, um I, can identify the software and use another badge who can't do its gradient.

D

So you'd complete work at the subgraph to create the gradient bottom, the input tensor and the output tensor. But then you'll use different input data or from a different tensor to actually run the gradient on yeah. So yeah.

E

I mean I, think I mean I, think that that a presentation is not perfect, but we I mean it is trying to retain some of the characteristics and invariance of the original graph. So, as I said, if there are eats really, this node is capturing two stages of computation. One is construction of the gradient sub graph and the second is the evaluation of the gradient sub graph.

E

If you, if you ignore optimization questions, the gradient sub graph computation can be thought of as a computation that takes input and label and produces the deltas, and that is what this set of inputs and outputs are trying to capture the fact that, ultimately, the sub graph, that produced is takes input and label and produces some deltas internally. The implementation may bar optimization, but this is simply directly use computation that goes on in the left like in the my inference, function and loss value will get me directly.

E

Reuse those values, but that's not being shown here as an implementation detail, but conceptually.

E

And function 6i, if you have a function f of the x1 x2 x3, we are producing a function that given the values of x1, x2 x3 will return like the Delta F by Delta x1 into the error and so on.

E

That's the mum mental model.

A

Okay, any other questions comments about this spot.

A

Yeah, okay, there's a little bit confusion, so maybe some additional explanation: some document could be helpful. Oh.

C

Will you help document and hoping against.

A

Okay, okay, so returning to the question about the additional examples that we could use for testing, did you get a chance to work on work.

C

On it, you think this to example, get up.

A

Overhead, the linear model and then.

C

A

What kind of model is that.

C

A

Right I know it's Jesus crucification, so is it a bigger Network is.

C

The audio network- it's not very big but convolutional.

A

C

Here model yeah, okay,.

A

Yeah I think it would be also interesting to see some recurrent networks, possibly maybe.

F

Hiiiiii we want to chime in here, I I, don't think we want to overcomplicate this, like we already spent a lot of effort trying to validate this and I. Think like it, I feel I personally feel it's time to bring this up to the community and hear the feedback. First before we put an extra burden on ourselves.

G

Two examples are, in the same key, are.

C

They they are not because, but I can add this way, only I'm, not sure if a guy is it because Emmet, if you want to check in a myth example, it means that you'll need to check in the house and a big binary photo file, which is not good, but what am I getting culture, cultural, advanced clothes for their time.

G

And also you give scripts to create that training model for viewing. Is that also in the same er.

C

C

My name I can share it from a good channel, but, as I said, they are not bad. They always there a couple tons of corner cases that this group cannot handle sure to do.

D

The channels fine yep yeah.

C

I can I should be able to open that file, but why why I cannot do it.

C

Okay, but anyway, we do have very detailed reference documentation for query operator.

C

Then try it again: they deepen 5l. You can go to put my dog's operator Stanley, okay,.

C

C

C

C

C

Okay, so you will see a simple square and how we can connect gradient into the imprint Square and get this recording of some operators. There are much more examples with visualization here. You can read them and discuss so we can discuss if you have comparable questions or something yes- and here maybe has a nation for the attributes you can see cui has been in full square.

C

We compute a query offsetting first leg forward of the interest, Square and use other inputs, particularly over the or join the original, is W, and here we have wloz code, one yeah. So.

C

Is read, read, read.

C

Okay, NATO and beta.

A

Okay, so last time we just discussed about unit testing for gradient calculator. We.

C

Do have two small test for learning: okay,.

C

Here so those two things are quite quite think: oh we just go. Wait, wait, wait.

C

C

Okay, so both tilting our builder can scale up. Scalar variables is one day. One of them only contains a single ad note and I compute the gradient by running some simple pipelines. Expression like this- and this test contains to note ed and Ahmad creation and yeah.

C

That's I, guess the most basic, as we have other than that I I don't feel it's possible to add every to end a temporary operator in a shorthand because it will require an auto differentiation engine which is a book.

C

Just to for basically I guess the list could set up other things.

E

A

So you're awesome your suggestions that just heron to unit test is sufficient or for a simple cases.

F

So the reasoning here is the the gradient up is a very, very special heart. It's in the sense that it can it's a it's a placeholder that can represent the gradient function of any graph, any graph, so it's impossible to cover in arbitrary graph in four four four gradients or unit tests.

F

So we reflected two simple example to illustrate the idea and expected behavior for two simple graphs example and.

C

Also, as always, the forward is career is well-defined, the vector is also will define as well.

E

Oh yeah I mean I Griffith, portulaca, saying earlier, I mean I, think it's good to push this forward and take this to the community and because even with the standard, onyx I mean infrastructure evolved over a long period. After the original thing, starter dominatus and until you know the proposal is accepted.

E

Why invest a lot of effort in other things, and unless it's going to be accepted in this form, I mean building lots of other things could end up being a waste. If so, I think it's better to push this and.

A

Your suggestions that will have a good chance of being rejected, no.

E

No I'm saying by.

E

D

E

It's not like tag testing, you know.

E

This contains, in other things before we take it forward. I think I mean it's. It's always going to be an evolving process, so I mean over time. More and more testing infrastructure will be added, but this is a reasonable starting point.

F

Yep agree, I think we should definitely move forward with this PR and invite the other tracks other sick groups to help reviewing this.

A

Time, raishin project is a tan white couple. People from Microsoft who wore from the I seeds related to this but and I did invite them, but they didn't respond. That guess so.

A

F

A

Really not next meeting.

F

Shall we just invite the guys from other sick group.

A

We do have a die from consultancy group, yeah.

F

I think we need.

A

A framework converter.

F

Unit test, yes,.

A

F

Take from these three tracks.

A

Till do have any comments about all this I.

G

Think sure we were sitting by others to do this together, we'll be good to run some sort of a quick demo. All this works and yeah.

F

You already have to spam. Alright.

G

F

Like the last time, are you sure that is a flow yeah.

G

But not now you've made some changes here using I.

F

What's the difference, I.

G

Don't I know what I'll see a few changes in this PR and we know.

F

I think that the fundamentals is stood out right.

D

G

Haven't looked into the details, so I cannot comment on that. Yet.

A

Okay, so now we have a do some example, which is with Emily, stolen.

G

Yeah, even we run your test case here, I had to show that it works. The gradient works for for add more nine for.

G

You really coach fire I'm, just trying to see a running code that works right. How.

C

Else for that you need I'd work or.

F

Yeah, so I think we still need to clarify own expectation here, so we are not trying to build a runtime engine. We are trying to define us back that can represent runtime for training, so line I still want to reiterate on a clear-cut like there are some jobs that that should be left on time and ring, or example, implementing the extra gradient up. That's basically the whole the essence of the whole training from work using our auto grad or Auto diff.

F

So our object is to confirm us back that can represent and describe a training graph.

F

G

Yeah yeah sure yeah.

F

G

F

I understand that, like with you, I would like you maybe, like you I, feel that you have the concern of whether this thing will work with their engine. Yes, that's because, like we don't have the binary that around the training engine that implement this well, I can tell you that our next one time is taking this back and we're implementing this. So.

G

F

Well-Tested idea and can represent the training graph. If that's your concern, yeah.

G

Yeah, another concern is how we can produce this graph right. We haven't I know you have scripts to write that is sort of the training angle and gradient operators, but which framework can even produce this graph. The.

F

Onyx, the Onyx trainer, the training framework, no.

G

No I'm not going to sell right. You start from some outlying onyx itself is not usually the starting point right.

G

They start with a model from some other frameworks that does exchangeable format if I already trained this in pathology or tensor flow. So how I can produce this Onix training.

C

So you need power, does light and the logic behind would be very similar to what is it good at.

F

Me yes, but I think that's why we need to involve the convertor guys too. Yes,.

G

Joe, that's why I think that's important to at least have you know my torch or tensorflow converter, to see this pack and and tell us whether they can produce this? Okay, that's my point! So maybe next week I'll come through. We have you know a Gunther I, think he's the guy you're taking control and convert that into onyx and I torch. I, don't know who is maybe bend it yeah. That is a way to make sure the right people are in our next meeting, so they can make your posture film comment on this right.

G

Yeah I think I believe with this Sun as I did some exercise so I'm. Looking at the training people, try to you know, do a trainable model, egotistical. That seems to be okay. Of course, you might have changed something here. I may need to look into the changes, but I don't like that, but it shouldn't be a big change. So so I don't see a big problem there right yeah.

F

Okay, I think so on the converse: si: let's invite spendin 4/pi torch and conservative flow. Yes,.

G

F

On the a unit head side, let's invite Ahmad I think who is driving the unit test, see coupe.

E

F

And on the framework side undo now, anyone who we should invite.

C

Maybe filled up with his own vacation yeah.

F

F

Yeah yeah, but let's Priscilla stood like I, was just sitting right next to curtain. I think we have talked to him about this proposal quite a few, sometimes so yeah, and he was part of the involved in this design. Okay,.

G

A

Okay, so you will give me the emails of the people who I should invite for next time right.

F

A

You will talk to them to make sure that they actually attend. Yes,.

F

A

If there's lovely tongues.

F

Will just lit to coordinate and make sure like all right, there's one representative from each track that attenders, okay.

A

F

A

Will be in two weeks right: yeah.

F

Let's try to target for two weeks: okay,.

A

F

And like what another thing on that, the schedule kind of so like I, know, onyx is trying to release. The point seven occur is that right, operation, police's.

C

For targeting this Thursday, yes.

F

So and then 0.7 targeting November, if I remember correctly so I I will propose like we should try to push this in the point. 7 timeframe, I think.

A

They wanted it to be about training, so if we're not ready, then they will have to wait for us to be ready. Yes,.

F

That's right, that's why I think that's our timeline that we see in our last meeting yeah.

A

And it also may be related to the next onyx in your submission. Yes,.

F

A

I would like to get information sooner rather than later, because we might need to get visas, etc.

F

Yes, cool so yeah. Let's follow you the other guys to invite and.

A

F

Meet in two weeks and discuss this okay.

A

Sounds good? Anybody else wants to discuss anything else.

C

If you, if the person from other working group, then result to you, so you can ping us, we can you ping, other micro, super people.

A

G

That's one coming on the PR, since you have this 2314 assaulting one, the others. Can you put them in some kind of state, or so you know they have so many I, don't even know. Some of them are really up-to-date, and can you put some kind of comments, or so it will confuse people right.

C

So I guess, if we want to close it with this single PR, because then I counted any feel that if I don't put everything to a place, no one knows what all picture it is and now so by principles. I will just go with this big PR and I will close out other one that that's great I support.

C

How to come in.

C

This is kind of different from standard PR, creating process, because the usually quite small changes and push it. But this not a small change and I guess. Everyone need to see everything together. Yeah.

F

C

Good, yes, yes,.

F

I think, like I, have proposed a procedure to be like this, so let's get a high level of grooming on this journal. One big PR, so that to make sure that our approach is correct. Once we have that sign up from the the committee, we can go back and work on each individual, PRS and work on the details and try to get those some OPRS type thing.

C

Who you still on, who has the video yeah.

F

So like that, that big PR is just it's just too hard to review and then for I think it's just mainly for getting a high level idea and therefore picture of our complete proposal. I.

C

Well, but so to to catch this PR I sent two days to marriage, equality and now we'll make some let Sarah changes to think them, because some PRS are they have interaction. I cannot just protect one in the you know another. If we want to merge those changes back in me, I cannot use it comments. I need to do some menu management. Can you get manipulation to my code.

F

So, are you proposing we just do a one-time checking for this one, big PR.

C

So so, basically there are three paragraphs in this PR at miter slot and Quarian related for work.

F

I'll be very happy if we can get this one picture, I really thought it might be hard. Let's, let's, let's go see and see how there was their opinion. How about that.

B

Okay: okay, yeah, that's good.

A

Now, maybe just add comments into the small appears that they're all nurse into the larger one. So if people are interested, they should look at the larger one perspective.

C

Yeah sure this is a query.

G

Yeah, it's going to be hard to take like small PRS and try the whole training out that you need everything together anyway,.

A

Yeah I hope that's good. That way tank was able to know Jason together and we should.

G

A

F

You for the hot works, that's a lot of work. Yeah.

A

F

Thank you. Okay,.

A

So ah I said also we just meet in two weeks and the vision gives me a list of people to invite red.

C

A

Okay, thank you. So much have a nice to it and see you.

C