Rust Programming Language Oxidize Global 2020, 28 Sep 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Using TensorFlow in Embedded Rust

Description

Some microcontrollers are powerful enough to usefully run Machine Learning (ML) models on audio, motion or even image data. Even though the practical applications of this are not yet clear, there is considerable interest in Edge ML.

TensorFlow is a popular framework for training and running ML models, and there are good resources for training and miniaturising TensorFlow models. This talk is an introduction to using these models in an Embedded Rust project, and briefly walks through the process of instantiating a model, setting input tensors, running inference and reading outputs. We will use a Rust crate that wraps the C API, and this talk will outline how this crate works and how it catches many common errors at compile time.

A

There we go from machine learning, to machine learning from a generative art tracks to richard meadows, talking about integrating tensorflow or using tensorflow on embedded rust, which I also find it very interesting subject, because we've been talking so much about integrating rust with other stuff over the whole morning in the chat and during the talks. So here's the talk that applies it give it a go. Richard.

B

Okay, so my name is richard meadows and I'm going to be talking about using the tensorflow framework in embedded rust. So tensorflow is a machine learning framework. So what is machine learning so? Firstly, machine learning is about transferring transforming some input data into some useful outputs.

B

This is something we do all the time when we're programming conditional statements, if statements, but in this case we're going to use a model to make the transformation rather than handcrafted code.

B

So what's more, this model is created using an algorithm, usually based on training data, because it's generated computationally the model can be much more complex than anything. We could practically create by hand, and this enables new applications.

B

So a little bit about tensorflow, you can think of tensorflow as a toolbox machine learning. It gives you many of the tools you need.

B

There are other frameworks, but tensorflow is one of the most popular and that gives it the advantage of good documentation and a big community. It also has support for embedded platforms, which is important for us.

B

Okay, so this slide is the simplest drawing of machine learning. I could come up with so at the top there's training, which is where we create the model.

B

The model then goes down the arrow and the model is used to influence, so influence is just a name for the process of creating outputs from the input data, so this drawing doesn't quite cover all possible forms of machine learning, there's also reinforcement, learning and other schemes with feedback, but this is good enough for today and something else about this talk, so we're only going to cover inference.

B

Training is something that would typically happen on a on a pc or a server with lots of compute resources, maybe also a gpu- and you can do this in python, java c, plus rust, of course, but either way at the end of training. You get the model.

B

There is one really important part of training called quantization and we'll make a small exception to talk about that later.

B

It's also worth at this point, defining the term edge machine learning you see as the acronym edge ml as this is where one or other of these steps happens physically close to the input data, so that's usually a sensor or microphone, and that means it's happening on an embedded.

B

B

Okay, so more details about using tensorflow and microcontrollers.

B

So this is a subset of the overall tensorflow project and the official name is tensorflow lite for microcontrollers, that's a bit of a handful, so I'm gonna shorten it to tensorflow micro. uh For the rest of this talk, you may also see some of the other abbreviations there.

B

One of the good resources to learn in this area is a book, so that's written by two of the contributors.

B

There's also some screencasts online relating to this book and all this material is a great place to learn more about training and quantization, because we're not going to cover those in detail.

B

Okay, so so far we know tensorflow micro is written in c plus and it runs on microcontrollers. So how do we go about making use of that in a rust project?.

B

So one of the features of rust is that it can be application, binary, interface, api compatible with c.

B

So in simple terms, that means that we can configure things, so we can call c functions from within our rust and it even works the other way around. So we can call loss functions from within our c programs.

B

So so, once we have that power to call c functions from our loss code, we can write what's called bindings.

B

So these are sets of rust functions, the core corresponding c functions, and we can group and structure those rust functions together. So it looks like a regular rust api, and that gives it a form and structure. We expect when we're using a rust api.

B

And once this abstraction is built and tested, then we can now we can use it without really worrying that there's actually c under there. So it's important not to forget entirely, but this is a good enough abstraction and it can take us a long way.

B

Okay, so to support different platforms. Tensorflow comes in several flavors, so there's three. I've got us on this slide, and this is about the code. That's running our influence step and um the three types differ on the platform that they're intended to run on and for each one there's a set of russ bindings. These are the rust functions that have been written to the call the c functions in the tensorflow project and on the right turn column.

B

There there's the list of links to the relevant crates and crate style here and um for running on microcontrollers, particularly um we're looking for crates that doesn't use the rough standard library, and so only the bottom line on this chart has that property.

B

So this is useful for microcontrollers and it's also useful for other platforms that traditionally haven't had the standard library or where it's useful to save the space of not having the standard library in your binary.

B

So wasm is a good example of that.

B

There is a rust special interest group within the tensorflow projects and that's primarily concerns around the top line on the on the chart there um and the rest bindings. There are sort of semi-official, um google doesn't give them full support, but but they are developed, particularly by some google employees.

B

Now the other interesting thing to note here is that tensorflow, lite and below it tends to flow light for microcontrollers. These have their own model format.

B

So this is a way of writing the model as a as a flat file format and we're going to encounter that in a few minutes.

B

Okay, so what have we seen so far, so the last few slides have been about machine learning about tensorflow and how we can run inference on microcontrollers.

B

So the next few slides are going to be about using all of this in a real life rust project.

B

So we're going to go for typical, rust projects where we use cargo.

B

So cargo is our build system and we add add the bindings crate to cargo in the usual way, which is by adding it to our cargo.tomorrow file, and you can find some details about the crate on crates that I owe.

B

And the really neat thing here is that this works out when we're cross-compiling as well, uh particularly um cortex, is probably the most commonly uh commonly used cross-compiling targets and the only difference to a normal busquet is that we need to have, in this case a c-plus plus compiler behind the scenes um so yeah. The rust ecosystem here is actually doing some great things.

B

There is one open issue I think that's been actively worked on, um but for the moment it does need a nightly build flag um because we use some of the same crates on both on the host side of the build system and on the targets which card which cargo doesn't support by default.

B

Okay, so now we're going to get onto the code so in this example we're going to work towards performing inference.

B

So we finished the training step by exporting the model into a flat binary file, and this is the file here microspeech.tflight to escape the training step. We can use one of the example models from the tensorflow projects and in fact this is one of those.

B

So we want to get the bytes from our model file, obviously into our program, and the include bytes macro does exactly that.

B

On the next step, tensorflow has some model validation and passing functions, and we run them like this.

B

So there's two other things that we need to start doing inference and I'm going to skip over them here, but I promise they're just one or two lines of code, so we need something for looking up the dsp operations that are used as part of the as part of the influence process.

B

The default option is to use all the possible dsp operations supported by tensorflow, and but the downside of that for microcontrollers is that it uses lots of flash memory.

B

So we can get around that by specifying only the operations we need at this stage.

B

The second thing we need here is tensorflow internally uses, has its own stack or stack or heap memory that it uses and it allocates to its own memory allocator. So we need to provide some memory for that.

B

Okay, so once we've got all those three things: the model, the opera solver and the tensor arena, we can put them together to make an interpreter.

B

B

So at this point we actually need to come back to what our tensor is so in mathematics, tensor tensor is used for various things. I'm not going to explain that here, but here we can think of the tensor as just being an n-dimensional array of data um and for putting it into the model. Actually, we needs to be a flat flat away and.

A

B

Some other methods you can call on the interpreter to find out what the what the dimensionality should be, which is fixed by the model, but in this case we has to be flattened down and we pass this as a slice and once that's done we're almost there.

B

So the invoke method actually runs the model, and this is what takes the time normally. So this is quite processor intensive and then, after that, we can get the output tensor.

B

So that's pretty much all there is to it. um If you want to see all this this code in an example, then in the tf micro crates, it's uh it's one of the tests, so you can see all this code making your full program.

B

Okay! So now it's time for a quick demo. um So so hopefully what you can see is a stm32 development board, and this is running one of the examples and the the output from the model, the output tensor.

B

In this case the dimensionality is one by four and so there's four output values and these are being plotted on the bar graph on the screen.

B

So you may also get audio from this. So hopefully it's someone saying the word no a couple of times and if the model detects that word as a keyword detection, then the bar will go over a threshold. I think 220 and the bar changes color to signify the phrase that's being detected.

B

So I'm just going to play the video.

B

And just once more, no, no.

B

Okay, um so this model- probably isn't the um you know, it's probably not the greatest model yeah, it's probably not the most robust, but it does work as a superficial level, and um so there we go.

B

um One interesting thing to note here is that the output data is, is eight bit integers, so it's over and unsigned in so the possible values are between zero and two five five, um so this has become because the model has been quantized so normally during training, the model weights and the inputs and outputs are normally floating point values.

B

um But at the end of the training step you can use quantization to reduce this to eight bits operations and at the same time, you keep track to make sure that the accuracy of the model isn't being degraded too much so when generating models from microcontrollers.

B

This is really important because it significantly increases the execution speed. um So you should probably you should do this when creating a model for microcontrollers unless you're specifically trying something else: okay back to the slides, okay. So let's step back a bit um and talk about the crates that we've used to do this.

B

So this is an open source, crate written by myself and kevin hill and it's on craits.io. It's documented. It's got tests, it's got ci. um It's partially checked against the rust api guidelines and probably could do with a little more work there. But it's pretty usable and it follows the usual rust patterns that we expect.

B

Okay, so challenges around maintaining equate like this, um so the tensorflow projects as a whole gets about 500 commits a week. So that's a pretty big velocity of change.

B

The tensorflow micro subsets is not so bad, but still there's several people working full time on this and the c api does change fairly regularly.

B

However, the good thing about ross bindings is they give us some flexibility and actually for some of the recent changes we've been able to refactor for the changes in the c plus plus api, but the rest api remains unchanged.

B

Okay, um so why do this invest.

B

So um some of the benefits of bust here is really the tooling, so not the language, but actually the ecosystems of tools like cargo. The cc crates and bind gen um that really help us out so without cargo.

B

Integrating build systems from a microcontroller vendor with tensorflow's own can be quite a task and there's actually various development boards that advertise having a tensorflow build system and hardwares and hardware examples, and they advertise this as a feature when selling real hardware and, of course, if you're trying to run tensorflow this, this is worth paying a small premium for, however, with cargo, um this is just a standard uh here we have a standard, build system, and this all comes for free.

B

So the rust system, the rust ecosystem, gives us this great, build tooling and we can easily add tensorflow to an existing project.

B

It also allows us very easily to build for multiple targets, so we can use the same crate for testing on a desktop pc and then exactly the same code then gets cross-compiled and runs on a microcontroller.

B

um However, to achieve this kind of composability of the library, the language also plays a role.

B

So if the language allows memory a listing, then we actually have to manually evaluate evaluate the memory usage, all the different component parts in order to make sure we're building a free program.

B

However, with a memory safe language like rust, we can delegate this task to the compiler. I'm trusting it to error. If there's a mistake.

B

Okay, so time for some codes. Okay, so if you remember that function a few slides ago where we created the micro interpreter out of the three component parts, uh so the model, the op resolver and the tensor arena.

B

So here it is, as a c function, definition uh taken out of the tensorflow project and um okay, that's pretty big comments, I'm just going to summarize it so it says that the lifetime of all these things must be at least as long as the lifetime of the interpreter object, and then it gives you some advice for doing this in a typical project.

B

So um this, if you've been programming in a language of manual memory management for a while, then you can understand this and you can do this, um but um so we want to make things more accessible and and also in the real world things change and um you need to keep. You would need to keep uh keep checking this invariant for all the time. Your project changes and that's not so great.

B

What lost allows us to do is is model model, this constraint actually in the language as lifetime annotations, so here's some must so this is a pretty big function. Definition um some moderately advanced to us going on here.

B

I'm just going to give you a few seconds to read it.

B

Okay, so this function says something like so we're going to implement the micro interpreter with its lifetime a and then the function is public function new and it has the lifetime m which exceeds lifetime a and the lifetime t which exceeds lifetime a and some generic types.

B

Then, when we pass the model into the function, we tell the rust compiler that we're borrowing the model for the lifetime m, which, from the previous line, we know, exceeds the lifetime of the micro interpreter.

B

So actually, we've now enforced that constraint in software, and that means that when we do a ci build and this constraint gets checked and that's really useful.

B

um There's an extra some more advanced rust going on at the bottom here, where the tensor arena can either be immutably borrowed from the stack or come from the heap. So if you're running a larger microcontroller projects, you might have an allocator for your heap and in this case you can pass either here.

B

B

So quick summary, so we can make use of tensorflow I like for microcontrollers in rust, through building and sharing rust binaries. We can make use of tensorflow and many other big c and c plus code bases that are useful for embedded projects.

B

Rust gives us some great advantages with consistent, build tooling, combined with a memory safe language, and we can easily compose a program with multiple complex libraries without introducing new bugs.

B

The rust language also helps maintenance, because we can make sure we're keeping up these invariants with the compiler and the compiler can check this in continuous integration.

B

Okay, great, so that's the end of this slide. I'm happy to take a few questions, but otherwise thank you for listening.

A

Yeah, thank you for the talk and I'm looking. This talk also sparked a lot of conversation in the speaker's room and uh I'll have to see how many of those questions are actually not asking other people questions.

B

A

ah Here, question for richard meadows: what kind of data applications do you recommend using ml on microcontrollers for so.

B

um So the models that we that I've demoed here are all um so they're all kind of linear models. So what that means is that the the input data goes through a series of transformations, a series of dsp operations um and they they form a kind of linear graph to get to the outputs.

B

um So this is good for things like uh classifiers, so um we demo here with speech classifier. um It can also be used. uh Some very basic image classification is possible um and also things like classifying accelerometer movements, so gesture detection.

B

um I think um there are other more advanced forms of um of machine learning. Here we aren't supported so there's, uh there's long, long-term, short-term memory and other kind of recursive structures that tensorflow micro doesn't yet support. um So we can't do everything, um so I think yeah. In short, we can do classification, um but I think big sort of large data processing is is not yet ready for micro controllers.

A

Thank you. I, unless someone quickly posts a question, I have seen no other questions to you but, as I said, a lot of discussion that you can hop into.

B

A

Thank you for that talk. Oh here's. Here's one question: um how limited is a tf light for microcontrollers feature wise compared to the regular tf light, which is already limited compared to the regular tensorflow.

B

Okay, yes, so I think part of that is what we're just talking about that that's things like the recursive networks and long-term short-term memory isn't yet supported in tensorflow lite.

B

um My understanding is that tensorflow lite for microcontroller supports many of the many of the operations that are supported in lights. So I think it's definitely a subset of the overall tensorflow projects, but many of the features are now trickling down.

A

Thank you. Okay, then, with that I'd leave you into the discussion and we'll see us after the.

A

A