Rust Programming Language Rusty Days 2020, 31 Jul 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Rusty Days 2020 - Lachezar Lechev: Drone Control - "Controlling a drone using Rust over WiFi"

Description

Agenda ► https://rusty-days.org/agenda
Slides ►https://rusty-days.org/assets/slides/04-drone-control.pdf
Playlist with all talks ► https://www.youtube.com/playlist?list=PLf3u8NhoEikhTC5radGrmmqdkOK-xMDoZ

Follow ►
Facebook: https://rusty-days.org/facebook
Twitch: https://rusty-days.org/twitch
Twitter: https://rusty-days.org/twitter

This video ►
Have you flown a drone? What about writing Rust? Let me tell you about our efforts to hack on a drone using Rust and build a demo project that will showcase some practical capabilities.

The drone we are using has a couple of ways you can communicate with. We are focusing our efforts on the WiFi (UDP) communication and decoding the Frames used by the drone using the open-source C SDK provided by the manufacturer.

We will share with you what were the hurdles we've encountered as well as what we are planning for the future of the project.

A

Hello, everyone and welcome to my talk about drone control or controlling a drone using rust.

A

My name is lachizad lechev. I know it's hard to pronounce or remember so, don't worry if you get it wrong or forget it, I'm from bulgaria and I'm currently working at addicts during the day, I'm using roast for the last one and a half years yay and it's full-time job. So this was my dream only two years ago and I finally achieved it.

A

Our work is centered around the ethereum network, blockchain ethereum blockchain at the moment, and we are building a display at network for reduced upfront and increased user privacy at night. However, I play around the ira rust working group that we have started at the beginning of this year and the goal is to help and push the open source community to the growing into the growing aerospace industry, by providing information, materials tools, uh crates and etc, to the industry itself and to hobbyists. Of course, one of the main reasons that I wanted to start.

A

This working group was in fact that I couldn't find enough information at a reasonable level that I can understand at my job and at the working group I make and build open source projects.

A

I love to travel a lot and combine combined with the fact that I'm location independent. It's awesome, however, due to the current outbreak. Traveling is not a certain thing anymore.

A

Combined with traveling, I love to go to festivals and parties, I especially like trans music. After after all, we have to blow some steam somehow from time to time, apart from being software engineers and developers, as you can also see, I'm a heavy user of emojis as well they're part of me, and I just like it.

A

Today, we're going to take a look at the pirate anavi 4k drone. The company parrot makes a bunch of drones. Some of them fly some of them, don't initially, when the whole project started ignition. One of the other co-founders of ira rust started hacking on a different drone.

A

It is called the jumping zoom and it's a remote, controlled two-wheeler that can also jump. However, we, as in I thought it would be awesome if we make it work for the flying drones and try to control them.

A

The picture you see here is one of the packages that the pirate company has of the anavi 4k drone. It includes the sky controller as well that you can use to control the drone.

A

It is good time to mention that none of this is sponsored by them in any means, at least not for now.

A

It is an accessible small drone, especially because of the price, and it's easy to work with, and why did we choose it in the first place?

A

Well, we chose it because it has a lot of features, including wi-fi over which you control the drone. The drone has its own network that you can connect to.

A

It has a 4k camera and the parrot company provides a c sdk to be used for the drones, it's open sourced and it's the main way. We find many of the things that we need in order to hack on the drone using rust.

A

The company provides other sdks as well for building android and ios apps. They also include a python version, but it's using the csdk and that's where the other sdk pi part comes into play.

A

Pi part is a python sdk and it's very very helpful when things don't work and you wonder why.

A

There is a simulation tool called sphinx. It has many features that I have haven't reviewed or used. Yet, however, since I don't actually own an enough 4k drone, it's very useful to play around with it and built the rust sdk with it.

A

There is also the free flight app, which is for ios and android. That gives you the ability to connect to any parrot drone, and you can use it even for the simulation. I tried it one day, but that's not where what we are here for we want to use rust to control it.

A

There is the skies controller that I already mentioned, and the list of features goes.

A

A

Now that we know what we are working with it's time to take a look at, how does it work.

A

The main point now that it all has to do actually with communication and the way we communicate with the drone, is over wifi, the same way as the apps and the skype skype controller.

A

The skype controller is a to your phone. It has its own separate, wi-fi, antennas and stuff, and I think that the phone is used for the video stream itself instead of on controlling the uh the drone and regarding the connection and the communication, we need to talk about three main points.

A

The first one is the handshake. After connecting to the wi-fi of the drone, we need to establish communication between our controller. That is our program and the drone.

A

The pirate drones use their own protocol that we are going to take a look at. It includes different mechanisms and things that we are still finding out how to work with, and the protocol uses what we call frames and we will see what they look like and represent in a bit.

A

This is all incorporated in the ar sdk eras crate, which is in heavy development stage right now. Aka work in progress.

A

And now, let's start with the handshake.

A

After we have connected to the drone over wi-fi, we need to establish the connection between our program aka, the controller and the drone.

A

Each drone has its own initial port, it's usually four, four, four, four four and we use tcp and make sure that we handle the handshake correctly, especially the response from the drone.

A

Through the established gcp connection, we send the request, which is serialized with saturday and of json into a json string.

A

It includes the d2c port, which means drone to controller port alongside the controller name and type which I'm not really sure how the drone uses and there are other ports for the video streams which I haven't bothered to check. Yet, after sending the request, we should we should get a response in json, which we deserialize again with zelda and sadly, jason.

A

We get a controller to drone port from the drone and its response, which is basically how the controller will speak to the drone.

A

It also includes a status field, and if the status field is zero, then we have a connection. Otherwise we need to try the handshake again.

A

The response provides other fields as well uh regarding other options, ports, etc. uh We're we currently don't use them and honestly, at this point, I don't know what they are for yet.

A

And by this we have a connection hooray and to get a visual representation of what I just explained.

A

I've included the request logs here that you can see, and we use the same four four four four four port: to connect the drone and send the request to the drone, as well as the field we sent to it.

A

And finally, we have the successful response from the drone. Again, you can see the controller to drone port and the status fields, as well as the other fields that the drone sends and now that we have this connection with the drone established it's time to talk about the protocol.

A

The protocol uses what we call frames and the first part of the protocol is the ping-pong frames, slash mechanism.

A

Then we have the listener that listens to the provided, drone controller port for incoming from the drone messages and frames and serializes them.

A

Then. Finally, we have the command sender which we will use to send frames to control the drone.

A

The ping pong frames and mechanism in a nutshell means that the drone sends us a pink frame with some gibberish data, which I actually found out a couple of days ago. Before that I thought it also combines an actual frame. However, this is not the case. It's just gibberish and the drone expects a response of that pink frame called punk, which we will see how it works.

A

When we talk about the frames themselves, the punk frame should contain the same data, the same gibberish data that was received from the pink and if we don't send any punk response in seven seconds, the drone usually disconnects the controller as in our program, and we need to do the whole handshake again.

A

The listener at this point does some basic stuff, which are crucial to the communication, the connection itself and debugging, and so on.

A

It uses a udp socket with the controller to drone ports that was provided by the drone during the handshake and each received message can contain more than one frame and we will see how we cope with this and extract each frame from the message.

A

The listener also deserializes, all known and unknown frames, since it's still working progress, not all frame deserialization is implemented, and this gives us an easy way to gradually decode each frame, either by using the c or the pi priority sdks and looking into them and trying to understand what each byte means.

A

At this point, the listener is handling pink responses by sending a new punk frame to the command sender, so that it can also be sent to the drone, as well as the so-called acknowledge frames, which will get to it between two in a bit which will get to it in a bit.

A

I've also included some logs from the listener to better understand what what's happening. As we can see, we receive some bytes from the drone, and then we see the serialized version of the parse.

A

Bytes yeah, we are going to take a look at each individual element in a bit.

A

On the other hand, the command sender handles the sending of frames from our program to the drone.

A

It uses udp socket as well, and we expect it to send frames from the controller to the drone on the drone controller port. This is the same port that we have provided on the request when making the handshake.

A

Right now it sends the raw bites to the drone. The actual serialization is happening on a different place.

A

And the logging again, it's straightforward. This example is for the takeoff frame, which will tell the anatomy 4k drone to how to magically take off and hover above the ground, waiting for further frames and commands.

A

The earlier logs that we saw from the listener were actually the acknowledge response from the drone for the take off frame that we saw in the previous slide.

A

Basically it it is saying that it received the frame and executed it in order to fully understand and know what each thing means we need to take cut the frames and, as we mentioned before, we can receive more than one frame from the drone, and the way we distinguish between every frame like shown here is the way the frame and its bytes are arranged.

A

I've personally found that it's easier to understand the frame by representing the bytes as a u8 or unsigned 8-bit numbers.

A

And this message can roughly be translated to this.

A

The first frame we have we can decode uh has a type data which means that it's a frame containing some data, the buffer id, is pink and following the buffer id, we have the sequence id.

A

This sequence id is based on each individual buffer id and it starts from zero when the count gets to the limit of u8 it overflows and starts from zero. Again.

A

This way uh we can send an acknowledge frame to confirm to the drone uh that we've received a specific frame, but we can also make sure that the drone returns an acknowledge frame that guarantees us that a command we sent is executed.

A

For example, we might want to know if the drone has actually landed and executed the landing command, otherwise it can still hover over the surface.

A

The next four bytes uh or u32 in a nutshell, is the length of the full frame. It is serialized using a little engine or the less less significant byte. First, the length includes the headers type buffer sequence id each represented by one byte, the u32 that the length represents. This is four bytes and the feature length of the frame.

A

If there is one and by this length we can split each and every uh individual frame. If we have received more than one in similar matter, we can decode the second frame and, as in the first one, and the type here is data with acknowledge, which means that the drone requires us to send and acknowledge for this frame.

A

Meaning that we need to confirm that we have received it, then we have the buffer id dc event, which means drawn to controller event, and this is an an event that the drone sends us to notify us about something happening.

A

In this case, uh it's sending us a calibration state that the drone is into and as we saw in the previous slides, uh the message uh had 35 bytes and we can see that the length of the first frame is 23 and the length of the second one is 12, which is a total of 35 bytes.

A

And with the current implementation implemented types, this looks like so.

A

To elaborate a bit on the call buffer, id and sequence id, you can see that the pink frame has a sequence id1 and the dc event also has a sequence id of one.

A

Each buffer id has its own counter. The next time we receive a pink frame. It should be with a sequence of two. The same goes for the frames that we are sending, as you can also see. Not all types are currently implemented and defined, and, although the pink uh in the pink frame, we see that the feature is unknown and with the value of 3, this is not used, since it's part of the gibberish date of ping and not an actual feature.

A

Value now, let's take a look at the type and the buffer id.

A

The type basically means that what type of frame we have, what kind? As you can see, we have acknowledged data. Most of them are easy to uh yeah to to understand, but some of them we still don't know how to use or yeah. We haven't looked at them.

A

On the other side, we see the buffer id, they are usually separated in two distinct categories.

A

From the drone to the controller, they are usually prefixed with dc, uh not all uh the ping and the acknowledge from sent with technology the last one uh they are not prefixed yet, and uh we can see that the punk is not also not prefixed and it's part of the controller to draw buffer ids.

A

These are the buffer ids that we're going to use to send frames to the uh drone, and we already saw the dc event in action uh when we took a look at the calibration state example from the previous slide.

A

Now that we know at least what the headers do and how what the the frame looks like and represents it's now time to take a look at the frame itself and build a whole frame which we can use to send to the drone.

A

The frame struct is pretty simple and straightforward. The bulk of the work, however, is in the features those features represent each and every command event setting state etc. That the drone understands some features include only simple values that we saw in the previous example.

A

Some of the features include strings and special parsing like, for example, the setting the date and type time of the drone.

A

Even the frame for sending piloting commands to the drone includes a timestamp that is specially encoded, that is in a specific format, let's say, and we need to parse it for both serialization and the serialization.

A

It is important to mention, though, that when we are serializing the frame we serialize the feature first of the frame, and then we can calculate the total length of the frame in bytes.

A

When we have the total length, we can encode it at the right bytes. In order for the frame to be fully ready to be sent to the drone, as we saw in the messages example, uh it's usually like the third byte to the seven or something like that.

A

For the actual serialization and the serialization of the frames we are using the crate scroll. This means that for each struct or enum, we have to implement try into context and try from context which gives us the ability to serialize and serialize the bytes, as we saw earlier.

A

This does require a lot of research and boilerplate at this point in order to map every single feature that we can have- and this is why yeah we still have a lot to implement and track down in order to finish the whole serialization and the serialization frames.

A

Now I was going to show you a gif of a working example. However, let's see how the gods of life demo, uh if they're yeah going to allow me to show it, live to you instead of uh actual gif.

A

So first we need to start the sphinx 2. It's uh it's pretty simple command. uh They. It does steal your wi-fi card uh because it needs to create the same wi-fi as in the drone. It uses the same firmware as the drone, and it also includes this dot drone file. That has some settings. You can tweak and the 3d model and stuff like that when we started uh when we started it's going to give us this gazebo ui, and we can see the drone right here in the middle.

A

Maybe I can yeah and if I rotate it, we can see the camera here. So it's pointing at the back right now for reference.

A

We can insert a 3d model of some building an apartment building and usually there is a video stream of the drone.

A

However, because I don't have a dedicated gpu, uh I'm not sure if it's going to work very well, it usually doesn't.

A

It gets to the yeah to the initial view of the webcam of the camera actually, and then it doesn't, it doesn't update until the drone has landed.

A

However, uh let's try it and see if it works, so we have the drone here. uh I have this example uh for taking off with the drone, and I should take off hover for a couple of seconds.

A

Then it should fly up and down and land, as we can see here, uh the logs that I showed earlier with all the frames and the actual drone now moving and accepting the commands we are sending.

A

Let's try the video nope, the video is broken so I'll, just close it, and we can see that we're currently sending the take down command and now we've just sent the landing command and the drone has landed successfully.

A

The drone is we're still keeping the connection with the drone and it's still sending some information, although it has landed, and we can still start it up again very easily if we send another takeoff command.

A

And thankfully, this worked yay demo time, yes, oops.

A

This is uh still work in progress pretty much and we have a long long way to go.

A

However, it was really fun to build this and we are exploring new things that we can make and do with the drone.

A

I've been in contact, for example, uh with the worst computer vision working group, and I hope we can make some post processing of the video from the drone.

A

We still have a lot of line to do, especially to figure out what is a good approach for the api uh to make it easier, more idiomatic to use and to control the drone and each aspect of the flying itself.

A

There are, of course, much improvements. It's just hacking at this point and trying to make things work. There are definitely improvements to be made, especially separating the frames by receiving frames and sending frames to make it more comfortable to work with them and built the correct frames, instead of sending something that the drone won't understand and some of the other mechanisms that I haven't mentioned here, especially around the flying and sending the commands and receiving outputs of them or confirmations.

A

We still have a lot of implementations to do all the features uh and um yeah. We need to be able to distinguish between the receiving and the sending because uh yeah when we receive an unknown frame, and we don't know how to handle it. How this can crash the drone, and we should know all the correct values for.

A

That I hope this whole experience is inspiring because we had a lot of fun building it, especially thanks to ignition who started the whole journey, starting with the the small jumping sumo, and then I just hey: let's try the flying drone.

A

We had a lot of frustration. Why does it work what's broken? Why does it disconnect, however, uh with enough time and uh how to say it with enough enough uh yeah when you, when you want to do something, and you want it really hard, you can achieve it. It's just a matter of time and this project is part of the aero rust working group. You can find the sdk repository below, as well as some other cool projects we are working on in our github repository and organization.

A

I've provided some links to some of the parrots tools, as well as the rust computer vision organization. If you're interested in that- and thank you for your time.

A

A

A

Oh, a couple of slides.

A

A

I am I'm open for questions, so please share them on whatever streaming portal you're. Looking this at.

A

Question on twitch, rp ruiz. Sorry, if I missed this at the beginning, but is this an os project anyway, one can collaborate. Yes, the software is free and open source right now and I have provided you in the last slide with the actual repository that you can get involved.

A

A

Just jumping into the discord server ask questions. There are a lot of people there, especially people with backgrounds in whatever are aerospace, engineers and material engineers.

A

We have separate uh yeah, separate groups of people working on different projects in different areas. Okay, another question youtube as big new. Oh sorry, it's big neo uh would it be possible to run rust flight controller software directly on the drone. Without the network round trip yeah, I haven't checked that probably it might be possible to flash some custom firmware on it, but at this point we're still playing with the wi-fi question on twitch x kage.

A

Do you welcome contributors without much experience? Yes, we do the whole point of the ira rust community.

A

Not only a working group, we are a whole community is to share knowledge and information on various topics about aero space and engineering and whatever space. Even so, it's definitely encouraging to see new people coming in and trying to bring them up to speed or try to help them find their.

A

A

Well, another questions: maybe I can return the slides that we've.

A

A

Youtube, boris kiran is their interest in space industry to use rust. uh Yes, yes, there is. uh There is uh quite a big interest right now, especially in the rust community. As far as I know on another conference on oxidize, they were talking about uh how they want to bring uh the rust and to verify the compiler or of this part of it uh to be used in uh yeah in space, because there are really tight uh things that have like verification of the tools.

A

Each tool must be verified that it does what it uh it's meant to be and- and it's defined somehow question youtube. Michael lazovsky, have you tried to write it in other languages, for example, c or c, plus plus ah no, I'm mainly working with rust nowadays, and I don't even know, cnc plus plus a lot.

A

However, the main sdk is actually in c, so it might be interesting to take a look at.

A

A

Okay, uh okay, so I need to pick somebody with the most interesting question. I definitely like questions about irrest, because the main idea is to share information and, although I don't know anything about drones or space, I can actually go somewhere and ask somebody and start preparing myself even for a career there.

A

So in that case I would like to.

A

Okay, t twitch another question final question: okay, final spartan, I want we need to have a compiler purely made from rust before we can verify it. Targeting llvm would be heartache to verify.

A

If I remember correctly, uh we don't need pure uh rust compiler to do so. uh There are subsets of the c or c plus plus compilers, that are verified.

A

As far as I know, and we can do the same thing for rust, um especially um given that we now have two back ends or front ends, uh keep forgetting with crane lift. We can take a subset of rust and try to verify it bit by bit until we can start using it in space applications and, as I said, I do like questions uh regarding ir worst, so, I would say x, say: cage.

A

I'll choose you best question.

A

A

A

So, thank you, everybody. Sorry, I'm a bit nervous talking in front of people. However, it was really fun and I hope you had some inspirational things that you found in my talk.

A

A