OctoML Apache TVM Community, 22 Apr 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Apache TVM Community Meeting, April 22, 2021

Description

Apache TVM Community Meeting

Agenda
* Introductions
* welcome @bohanhou1998 as a new reviewer
* welcome new reviewer siyuan feng
* welcome new PMC member @kevinthesun
* Announcements
* May meeting will be devoted to AOT user facing API/interface design and implementation
* Go over 4 open RFCs first, to make sure everyone is levelset
* Then review needs/use cases for public user facing AOT API
* Meeting Tick/Tock
* Start alternating between EU and APAC friendly times for meeting
* Next meeting will keep same time, following meeting TBD
* New RFC Process
* https://github.com/apache/tvm-rfcs/pull/2
* Tensor IR
* Catch up on current status of Tensor IR implementation
* Open Discussion

A

Now, and uh just so, this is uh for for people who are not able to make this time slot um and for those just tuning in we've, just done some introductions and talked about the may meeting being aot uh focused. So look forward to that. The um we talked about the new rfc process, that's looking to be merged soon and uh any any other announcements or discussion topics before we jump into xeon's presentation.

B

I wanted to ask jason um regarding the iot, so are we focusing on one specific rfc or on all three or four that are currently out there.

A

That's a good question. uh What um I think we're we're we're going.

C

To put to either essentially we, which are the four, the four rfcs I am just.

B

So to be yours, then there is to my knowledge to st micro right. Okay,.

C

B

uh Tto munich also has uh one or two proposals which go in this direction.

C

Okay, I was not aware of this this last one. I I knew about this tea and then you know, but the one that I published, but I didn't know there were other proposals: okay, okay, I'll, look to them, but I mean yeah.

C

The idea would would be my idea at least was to come up with at least opinions about the interface which team, which is where I guess there are. The different use. Cases for different people, like st will have their own use cases, and we will have our use cases and stuff like that, so that that's that's going to be a bit controversial, so it would be nice to have and face to his discussion uh about the the public interface.

C

So what we want to expose to the average user depended by the rfcs uh that that's yeah but yeah. Of course we can. We can discuss uh on the specific rfc.

D

It might be nice just to have someone give an overview of the four rfcs at the meeting so that everyone understands.

E

D

They are and where they're coming from so these like have a have a higher level overview before diving in to say the interface or onto something else, so that there's a just kind of like a general picture of what the efficiencies are out there and it doesn't necessarily need to be a deep dive, but just to kind of give everyone an understanding of where everything is and so that we can all be on the same page.

C

Yeah, it makes sense, it makes absolutely sense. I I see what you're saying yeah it's uh it's absolutely it's absolutely it's the same thing to do to give a bit of overview, what different people are working on and to yeah and then to discuss the interface yeah. I agree.

B

It's really great, especially in this topic, it's really hard to keep up for me, it's almost a half day job just to keep up with all the postings yeah. With regard with regards to the aot.

A

C

Is exactly he's right.

A

Which is exactly why I think this is useful to have a higher bandwidth discussion, so great feedback.

A

I guess we'll work to find a volunteer to go over the four rfcs um or maybe a combination of volunteers and then uh to to dive into giuseppe's user-facing api. um Was there a particular one you wanted to go over, michael or michael? Is that good for you?

A

That's perfectly fine, as you have written down, okay, perfect great any other topics before we jump in to suns.

A

Awesome well soon I'll leave the floor to you and I'll stop sharing here and really looking forward to hearing about the latest updates on tensor ir and upstreaming work.

E

Okay, thank you. Jason and general jason told me that there is a meeting today, so I I have no time to prepare a presentation and a mix, and I didn't make any signs. Oh.

A

E

A

You're also really pretty quiet, so if you could maybe move the mic closer, if that's at all possible that might help okay, your audible is just uh a little harder.

E

A

I can hear you I don't know if ever anyone, okay.

E

I will change a mac.

A

Oh that's better. There.

A

E

Hello is that okay.

A

Perfect thanks.

E

A

Much better, yes,.

E

Okay yeah, I just changed my microphone.

E

Okay, so should. Should I give a big picture of the tensor area. I think I think I need to do it.

A

Yeah it'd be great to walk through the you know. The higher level context uh for for people like myself, who are coming into this.

E

Yeah, so I can you see my screen. Yes,.

E

Yeah it's some slides I make before so I will talk about the. I will talk about the tesla briefly, so that everyone knows about the big picture and the motivation and some key elements.

E

So the tesla, the basic motivation of the tesla, is to solve the problem of the auto test isolation. So today there's more hard work, hardware, instructions or we call the tesla intrinsic. So that's we will need to use some schedule or automatic schedule techniques to build to make full use of these accelerators instructions so that we will see. There is some elements of testosterone program. It needs optimized, loop nasting.

E

Of course we need that every tensor program needed with nasty and also the key idea is uh the key element is the multi-dimensional data, the multi-dimensional data loader and some multi-dimensional intensifies. The computation.

E

So here is the example of the tensor code, each instruction load, six by 16 16 by 16 data and do computations for 16 by 16. So it is different from the normal program that usually uses scholar, computation.

E

So here's two existing work and the last one is the current tbm just make the scholar body as the first first citizen, so that we, if we need to use a high level high dimensional intense intrinsic, so we just need to match the scalability, and that is a difficult thing for us to match. But uh right one is something like a firearm doesn't make top dog and the problem is also. Obviously it cannot.

E

It can only handle some simple, simple workloads, such as the metamorph, but it cannot generally use some intrinsic or the the other hardware instructions.

E

So the 10, the basic idea of the tesla, is a divide and stop all. We call divide and conquer just to make the outside luminescent and tensorize the body separate, so that we can just optimize both of them independently. What does that mean just to see the this?

E

Is this program a simple matrix mathematician and we can just make it as two parts: the loop outside loop, nasty 16 by 16 by 16 and the tensor as the body four by four by four tends by the body, and we do don't care about how the tesla's body computes we can use a scalar unit or we can use some instructions if both are okay, just makes it as a unit.

E

It's a whole unit. So we call it a block. A block will give some reason region and it will do something like uh it will.

E

Just compute see, use the data from a and b buffer, a and b, and we don't care how it do this computation, and we just make sure that we give you give the block the data and it will return the answer so that we will just uh make the problem easy. First, we will uh often optimize the outside luminescent so that we can just don't care how the block does and just a split fields and something like all the usually optimization and also after the loop nesting optimization. We can care about the block.

E

How we do some. uh Just micro, you can call the microkernel or block it will, how it computes a 4x4 by full metamode. We can use some tests like the body just like. If we have accelerator can do this, we can use the tensor intrinsic or if we do not have it, we can just call back to a scalar computation.

E

This is our key idea to make the uh tensorize the body and the loop nastiness independently.

E

So it's a just an example.

E

Also, there's some design details, design goals, but the zero design goes is just to make the tensor as a computation as our first class citizen, and we do not only use we can. Oh, we can not only use the scala computation but also use some uh tensor as the tensorize, the computation just like.

E

We can make a four by four by four block as a schedule units or we can just make us 16 by 16 by 16, just like a tensor core as a schedule unit, so that we can just match the uh shape of the tension intrinsic at the first step, rather than just a rather than after protein, stop split fusing and then the acid cancer is at the last step. We can just make the tensor access to the first step, which makes us much more easy to match or something replace the intrinsic.

E

Also, just a the second design goal is the just isolated internal computation, the tensorizer body for the external loops. We use some block signatures to show how the block computes and what does it, what data it needs and what data it produce.

E

The block signature basically contains just uh otherwise something like we have the iteration domains and tell the blocks how many iterations it needs. Just that and what's the iteration type, maybe the data dependence data parallel or maybe it's it is a reflection axis.

E

Also a block. The important thing is the block have its produced and the consumer regions. Each block will declare that how how many regions is wheat and white for this information we can just build a dependency graph from one block to another to others. Okay, so with these signatures, we do not care the the black bodies. If we just do a scheduling outside the block, we can just make a block as a black box.

E

The only information we need is how it iterators iterations or how is how many datas it consume and produce.

E

And also another thing is that we just make the block as the basic. So what does that mean? A block will contain the iteration domains that it need. We will just. We can just declare a block to declare the compensation on this slide. We will talk about the te and something relationship between t and tr and tessa.

E

In te we just use the shape to declare how many, how large the data is, will consume or produce, and in tr we just make a block. It is a block iteration and we will just make this 6 64 as the iteration domain.

E

With this signature, we can just uh complete the lupine ac. It is a also it is a default loop, lasting with 16 to make sure that the tensori is the correct function.

E

Also, um I I have.

E

I have told that the block is the scheduled unit schedule unit of the tensor, so as we can just make a block rather than current stage in here we have stage and uh the op, but in tensor we only have block a blocker is the schedule unit? So as we just can, we can just uh make a blog copy, for example, a computer or something like a computing line which has a makeup block in line or can be computed as something else somewhere else.

B

Sorry, can I ask questions.

E

Yeah, of course,.

B

Can you explain again why the inner block block b is not schedulable.

E

Yeah, of course, the block b has the uh let's look at the block b. The block b contains only one intrinsic instructions. This this line.

E

uh Sorry, if you you can see my mouse. Yes, I can oh yeah. It is the only body instructions right.

E

If we just look look at these instructions, we do not know how the instructions that's a computation, it can read the whole. It can read the whole region of a and b or it can. It can, with only one element of a and b. We do not know so that it is a non-schedule block, but the block b is not scheduled.

E

That means the block be in inside the block, be it is not scheduled, but if we look at the loose block, we can just make the block be compared at the block c.

E

Give me my idea, yep.

E

Just like some hierarchical scheduling. We can make the block b as a y unit. We can just move the block or split to fill the outside lubricants in I, but we cannot change the inner uh inter block. The inter body of the block b, that is called the block b, is not scheduled.

F

Okay, yeah just to add something. um The basic scheduling unit of ksir is a block, a block. What is the block? The block is an isolation unit that we can view as a black box. So there are some tags on the black black box saying. Okay, this black box will write the region of like some buffer, some regional some buffers and the way it writes is to using export some iterators for others to manipulate.

F

For example, here we have a like spatial iterator, which can be parallelized or reordered, and the scheduler do not necessarily have to know the inter, like the inside information about block. For example, in this case, block b has a very magical uh intrinsic on my fancy, vector at one. um The scheduler do not necessarily have notice. This is this is opaque to the schedule. So, even if block b is not scheduled because it is opaque, we can schedule like the loops on top of block b like loop. I.

B

And okay, so so it would be possible to unroll loop. I, for example,.

F

B

F

Definitely so um a good um a design design choice of having an isolated block unit like we can. Actually, um we can actually like if we have a micro kernel like we have um micro kernel of doing two benchmark modes of using some using some like asm assembly.

F

We can actually just isolate it in a block and we don't have to know what is inside, but we can schedule it outside.

E

F

E

E

So, uh just like add something if we testifies the block. If we testifies that in the tester we can still use schedule, the tensorized part, it is the new feature of the tensor, so we can just test right at the first of the schedule.

E

Okay, let's continue so just let's talk about something like transformations all we call its schedule.

E

The only thing we we are interested is the block box signature, including its iterators and its access regions. It's called access, regions, read the right regions and we doing transformations. We do not care about the black body. I have repeated this for so many times, so this enables loop transformations of the tensorized computation body just to make up. uh We can cancel as a block and we can still uh transformation transform some outside luminescent off the block.

E

The validation is not so important, not so important, but just take a breath see that's a block contains something contains the required iteration domains and we can use this to check if a tensor is valuable. For example, this k, the the x terms of k, is 16, so it is correct, but in this example the crack the accident of ko, is only eight. We should mismatch the requirement of sixteen, so it is like one tens area, so we can just make.

E

We can just use something like the iteration domains to validate whether it is a correct answer, but this is, it is not so important for us for us.

E

Yeah also, the block information itself contains, so we can only with use the simple block and we can complete the default for loop now scene because we have required the loop outside loop nesting regions.

E

E

I think uh it is something I could picture the big picture of psrr, so let's focus on the. uh So before focusing the upstream plan, anyone has questions.

A

One question I don't know: if you're going to get into this is backwards compatibility and uh you know if what what the status or changes looks like for. Anyone who has tensor ir kernels or you know the tensor ir passes and how that interfaces with the new block system.

E

Okay, good questions: uh for now we just use the we can use the hybrid uh sorry, not hybrid, just a tbm script as they import as the input of the tensor l. Also, we can just use the still use the te to import that cancer. Just like this, we can just use the left t exp tensor expression to build that tensor on the right.

E

It is okay, so we still can use the relay to to tow pi to t and to the ir and build on the devices it's okay and as for the parts we just lower test ir to the uh stage after the storage flatten. The current part, there is a positive current in tbm called storage flying. We just lower the cancer at the stage of after the storage flexion, so we can reduce the rest of the process.

E

Also, we can in the future. We may rewrite some particles and separate the original tbm ti and our tensor app. It may enable some new features, but as for the back compact compatibility, we still can make the tensor act can be built.

A

B

I have another question, so um maybe a more fundamental one. So, for example, on this slide, you could have on the left te on the right tir. So I'm not 100 clear on the terminology. Is tensor ir like a python interface, to create a tir graph, or is it somehow an extension of the tir language.

E

uh I think it is extension of the tr language. Part of the tesla is based on the tr and we still introduce some new elements. New data structure like block and the test, the imposing of sensor is enables an ir to ai scheduling, as we can see that before we just use the schedule 3, we do. We do scheduling and apply primitives on last schedule, 3, so that, after all the scheduling operators, we lower the schedule chain into tis right yeah, then the tesla is that ir to ir scheduling for each primitives.

E

We just input our tensor tensor, l and output tensor, so that we can print the tensor or get the tensor out at each of the scheduling.

B

Okay, so it's more than an alternative to hybrid script or ir builder. It's really a super set of the tir language.

E

Yeah, it's super set of tr language and also a super set of the currency scheduling. Okay, it's not only, uh but also, I always all this scheduling.

G

Yeah, so I think a good way that I have been conceptualizing, it is te is sort of has been an interface for a long time. The idea is that you use it to write a kernel, but te is not the ir really like the first thing you do. Is you take the like t stage graph and you schedule it down to tir? The problem is it's sort of been a one shot process where we expand the te to tir with a fixed set of primitives and there's no extensibility?

G

um In order to address this, I feel like what suwon and dunroe and a bunch of other people working on is they were taking tiara today and tensor ir sort of tir 2.0. The idea is, you add some new constructs that'll like improve schedulability they're, going to change the scheduling api to be more flexible, so you can add new, scheduled primitives um and scheduling in this case sort of a tir to tir function.

G

uh You can, you know, wrap them up and build new scheduling, primitives and then improving the compiler pipeline to take advantage of the new features, and I think, you'll, get a subset of the ir is what c1 is saying after storage flatten, which allows you to kind of keep backwards, compatibility for code generation or anything else. It sits at the back of the compiler.

G

uh If that makes sense,.

F

Yeah great point jared, I just some a few things to add out. So there are three pain points right now on cancer expression, and the first point is extensibility like right. Now we're relying on a very restrictive schedule tree and every time we want to add a scheduled print. It's super painful, like look.

F

Splitting conditional loops building is not working really properly right now, so, when we, when we like encounter like those imperfect, looks, there will be performance loss and the problem has been there for years, and um the second point is um tensorization like right now, t c is actually the the primary author of tensorization in on te2, so he has a lot to talk about. um Characterization t is a bit ad hoc.

F

We cannot schedule those parties fully testorize it afterwards and not quite flexible and and the third third point is like um what is the third point? I don't remember.

F

The third point is um right. Now um we only support scheduling on so those tea like very regular, like um gather style, uh yeah gather style computation.

F

So what if we have a sparse workload that we we want to tune up on, like those parts workloads might be just written in hybrid sql format, and we are unable to tune it with our current system and with tr. This is uh scheduling is basically just ir to error transformation, the simplest ir tire transformation and we are able to tune those irregular workloads with our like auto tr infrastructure.

F

Without too many without problems.

E

Thanks yeah, thanks to you any problems or any.

E

E

Okay, let's move to the current stage. We can see the upstream stage at this rfc or github issue.

E

We have there's two major milestones: uh yeah two major milestones. The first one is the data structures and some um some build uh build uh infrastructures and some schedule infrastructures.

E

The core data structure is called m1a and all these are finished, including the box structures and the the new tvm tv script process, and the m1p is just about the scheduling which are majorly done by you. I think, and it including some schedule, states, schedule, data structures and, and also some eta format updated by gohan and also the current stage is at the m1c, I'm going to say it's about how the wiring tensor up to the flattened ir.

E

So it contains some part. Some passes lower the in innate blocks and lca detections, and something like this and the last step is the buffer factor. I think uh all these codes are ready and we are running our inter internal code review and I think uh m1c can be done in uh in this week or early last or the next week. I think so. The next, I'm sorry would you like to zoom in a little bit. It is the phone okay, okay, uh maybe it's because my screen is 4k.

E

Thanks, okay, I think it is clear.

E

And the last step is m2a. M2A includes the schedule primitives like the split fields, computers, computing line, and so many others note that our primitives are decoupled so that we can write each of the primitive dependency.

E

It's also a new feature of the tensor that we can just make much easier to add a new primitives.

E

Oh okay, um after the two milestones, m1 and m2, uh we can use the case ir to build some workload, such as the convolution or metamorph.

E

And we are um we plan to finish it in one month or at least before july, yeah, nice, that's the current stage.

F

Yeah, after the infrastructure part is almost done and except for m1c, some, uh which is half a class away halfway down to. We are doing some really rigorous and harsh internal reviews. Right now, um after after the infrastructure is done, we can like submit those schedule, primitives separately in escape, like parallel way, because we do not have to deal with schedule 3 anymore and and if anyone is interested in adding new schedule, primitives like store ads or conditional, we are more than welcome and yeah.

E

Welcome everyone to review and develop ps and also add new, primitives or add new features to the tensor area.

A

Cool one question is uh documentation where I guess is it after m2a that you plan on putting together more documentation around how to use this and architecture.

E

Yeah, we will try to finish our documents after the whole m1, so it may be done by after my c merged.

F

Yeah- and um we are trying to finish a research paper too, on this topic um and also documentation, is really a strong need in our in our community, and we are um we really care about domination very much and welcome to review and rephrase and help revise. Thank you very much.

A

Create any other questions for xeon.

B

So, with regards to the scheduling or the for loops, basically so with tei trite wants to have a different uh for loop scheme than a rectangular pattern. Is it somehow possible to change to like go in a triangular pattern through a tensor through an array using tensor ir.

F

Just to confirm is that a triangular.

B

Pattern or rectangular pattern, what I mean is, if you have a double for loop and in a for loop, uh the the variable start or end variable is dependent on the running variable of the outer for loop. Is this possible with uh tensor ir, so I haven't seen this nte yet I see um so.

F

um So a um one b we have several prs for arithmetic analysis, um basically like a um integrated analysis on semi-fine um semi-fine maps and as long as the I think, the block is schedulable as long as long as um those um block.

G

Bindings are in semi-are fine. Well, I think, there's two questions right because, like you can always write the triangular pattern at the tir level, but the question then is like: do you want to schedule it as well? If you don't want to schedule it, then it's really easy and you can do that today. Even uh it might not be convenient to do yet, like I think, there's still some work on the convenience factor but like in tir, you could construct a fragment of ir. That does that um I think genre's point.

G

Is that the way the schedule matching works is it requires some constraints on the arithmetic to be able to do the matching and scheduling? Okay.

G

If I don't know, if that doesn't answer your question, we can uh discuss it more, but.

F

So it requires like basically like similar to political models that use a semi-semi-fine relationship between the variables and block variables.

A

And if you write a snippet of tir, that's valid tier but not schedulable, then does that mean that cogent still can function? But you just don't get the auto sensorization or many of the passes.

F

If, if there is a block there, which is not in that format, which is not in semi-fine uh map and the block becomes opaque and we cannot schedule a single block, but we are, we are still able to schedule those loops on top of the block and other blocks too.

F

So, uh even if it is not schedulable, we can do a code generation without a problem. Okay,.

G

Great yeah, my point is like the idea is to have an escape hatch in this case, where, like you, can write tir code, that does what you want, um but like the the polyhedral model or like uh even the scheduling, there's constraints, because, as you get the arithmetic more complex like you need to be able to analyze your arithmetic to sort of understand the geometric relationship between the variables.

G

If you will um and then you need some kind of like theory and constraints on what kind of arithmetic is allowed, where otherwise, it's very hard to automate the transformations um polyhedral like traditionally a lot of compilers just kind of give up. um You know if you step outside the polyhedron model, you you get stuck and there's some workarounds and stuff, but I think the idea here is that, like the opaque blocks at least allow you to have an escape patch um as you're working on some of these kernels.

G

um But then the scheduling is another question like. If you want automation, you have to fit into a narrower api and I think one of the other things generators talked about, which I don't know. If anyone talked about this morning was kind of um in generalizing the te patterns away from just maybe scatter, or something to provide other primitives that have nice scheduling, behavior uh based on like kind of patterns that people want.

G

So I think that's something else we could consider as we go forward, is maybe new primitives that have like um you know, sort of specialized scheduling and then maybe there's a triangle primitive or something. um I don't know what the use cases people want most are, but.

F

Yeah, so the basic idea here is that block isolation, just which means like, um even if it is not a scheduled pattern, it might be a scatterplot on other patterns. We just um we just put the right tag on the block like which region that they read, which we're going to write. What are broad bindings. We can properly schedule it, um no matter what pattern it is thanks.

F

A

A great question any other questions. I know we're over time technically, but it's a great discussion.

B

Maybe one last thing: um I think chris mentioned last time that there I mean it's more, an organizational part, not tens ir. So chris mentioned last time that there might be uh some subgroup meetings for different topics. So is this still in consideration.

A

Yeah, I think it's something we've talked about quite a bit. I think we're looking for a little uh more leadership or interest and and people stepping up to say. Yes, please uh we want to talk about xyz.

A

um So definitely if, if there is interest, let us know and- and we can either uh help someone step up or we can potentially kick it off on our side.

G

Yeah, I think the other thing that would be good from my perspective is that, because we're spread across so many time zones, um we could maybe move some of the meetings into like depending on who wants to attend into slots that allow more variants, because I know like if you're in asia it's what times. Work really well are different. If you're in europe, it's different uh like, for example, a bunch of people in uh octo ml or like west coast cal, you know california, seattle, people and people don't like getting up early.

G

So I think if we can find pairings of people that can find scheduling more independently, two that might help, because then we can talk more um without having to like get a global hard sync on everyone's time zones and um especially because I know there's some pairings, like you know, andrew's been working with the armed folks on um you know, embedded stuff, and you know I think, for example, a lot of people doing the auto scheduling are in asia, and so I think we can kind of that.

G

That's one pro that I I thought of uh in splitting off into subgroups.

A

Michael, do you have a particular request or idea there.

B

I think I mean the two things I'm currently interested in are the aot and tensor.

G

F

G

Maybe we can yeah, maybe we can talk about it this week, like offline on discuss or something about having a tensor ir meeting group as things start to get upstream, because I think there will be a lot of interest once these last couple pr's land um in terms of like using the new capabilities. uh I know there already is at octo, and I assume there is from other folks outside of octo.

A

And I'll record those and uh I'll talk to chris about them and we'll try to get some discuss uh and do you think that frequency is good, michael in terms of monthly cadence or or would you like to see these subgroups meet more commonly.

B

I think the frequency would be kind of okay, so it okay.

A

Really depends.

B

On the community, I think.

A

And we can discuss that on discuss as well. Okay, any other topics, great point, michael that thank you so much ceon for uh staying up uh past midnight for us here and uh really great content. Looking forward to seeing uh sensory, our improvements continue to land really great work, and with that we'll.

G

A

We'll go ahead and end it here and I'll post the video online and we'll link to it from discuss topic as well, yeah thanks so much guys. That was fun. It was good to see the overview great to see everyone take care.

B

Bye thanks bye.

E