Rust Programming Language RustFest Global 2020, 23 Dec 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Vivian Band - Rust for Safer Protocol Development — RustFest Global 2020

Description

The Rust philosophy is to use the type system to catch program design flaws at compile time. We aim to do the same for network protocols: we generate a typed model of protocol data, from which we can automatically generate protocol parsers in Rust, and can use these to validate the protocol design.

More at https://rustfest.global/session/11-rust-for-safer-protocol-development/

A

Vivian wants us to be safe and our code on the web to behave, use rust to generate code that will validate risky inputs, no need to be brave.

B

Hello, my name is vivian band and I'm a second year phd student at the university of glasgow studying network security. I was an intern on the improving protocol standards project, which is still ongoing and is led by colin perkins, with stephen mcquiston and dedges jacob as research assistants, so improving protocol standards.

B

The internet engineering task force standardizes network protocols. These are initially presented as drafts to working groups and then become official standards after further review. However, even after all of this peer review from several different sources, mistakes still sometimes appear in these standards documents, for example in the image on the right. The ascii diagram claims that the field option relay port is 13 bits. Long and option line is 19 bits long, but the text description says that they should both be 16 bits in length. These errors and inconsistencies create ambiguity for people implementing protocols.

B

What the improving protocol standards project aims to do is create a machine, readable, ascii diagram and text description format to allow us to detect these inconsistencies much more easily. These are minimally different from the format used in existing diagrams and descriptions with authors needing to use consistent label names and certain specific stock phrases in their descriptions.

B

These machine-readable documents allow us to build a tight model of protocol data. We call this custom type system developed as part of our project network packet representation network packet representation is programming, language agnostic, so we can use it to generate outputs in any language that we like.

B

I had used rust earlier to implement a bare bones version of the quick protocol a few years ago as part of my final year undergrad project, and I was impressed at how much safety added to systems programming. So rust was my language of choice for this internship project.

B

Our our first automatically generated parser libraries would be rust files because we wanted our resulting parsers to provide a good level of type safety.

B

Okay, so, first of all, let's take a step back and take a look at which types we need to describe network protocols before we can start building parsers and parser combinators. We need to know which basic types will need to pass in the first place. This internship was my first experience with uh with parsers. I use a lot of analogies when I'm trying to learn new concepts, so I like to think of these basic types as lego, bricks, which we can use to build cool, more ambitious structures later on.

B

There are several basic types that we can identify from protocol standards documents so we'll take a look at a tcp header as an example to demonstrate this fields which contain raw data can be modeled as a bitstream, so source port is just a 16-bit unsigned integer, that's just raw data fields which could contain one or more of the same element could be modeled as an array.

B

Some fields only appear under certain conditions and rely on values from other fields within the same protocol data unit, in this case the tcp header, to establish whether we're using that or not so we can call these constraint types since the they need to be checked.

B

Some fields require information not contained within this packet like an encryption key or a value from another protocol data unit somewhere. Within this draft, we can hold this information in a context. Data type which can be accessed by other protocol data units, which also feature in this draft when required, a field which can take on one of a limited possible set of values can be modeled as an enum, so we've got them here. An enum is indicated in these drafts uh with the stock phrase.

B

A tcp option is one of so is one of is the key phrasing where we need to use in these modified standards. Documents, packets and protocol data units as a whole can be considered as structure like types given that they contain a sequence of field types as constituent members.

B

One data type which doesn't feature in tcp but is integral to quick, would be the function data type, so these are required to perform conversions between different protocol data units. In this case it would be encryption and decryption of quix protected packet headers.

B

So we've got seven types in total in our network packet representation system bit strings. Arrays structs constraints, contexts enums and functions so now we get to the fun stuff automatic, rust, parser generation.

B

So we've got our basic building block sorted out now. How exactly can we use them to build more complex, parser combinators in rust?

B

So let's use that bit string example that we went over in the previous section when we're explaining our uh our custom types. We can automatically generate this custom rust type as a wrapper around an unsigned 16-bit integer in our rust output file pretty easily immediately after that we can generate a non-based parser for that type.

B

So this is a little bit more difficult to generate. There's a lot going on here, so we'll highlight a few key details. First argument for all of our parser functions is an input tuple. The first element of this tuple is a borrowed array of bytes, which for protocol testing, would be an incoming packet of some sort. Our parsers work at the bit level, so the second tuple element is an integer which tracks how many bits we've read in the current byte.

B

Our second argument is a mutable borrow of our current protocols. Context instance, since we might want to update this as well as read from it, our outputs are a non-specific result, type containing the remaining bytes left to be parsed, an updated bit counter and a tcp header item instantiated. With the correct value read from the byte array.

B

We also return a mutable reference to our possibly updated context. The parser function itself takes a defined number of bits from the input byte array. In this case it will take 16 bits and then assigns the value of those taking bits to the custom bus type as needed.

B

The order in which we generate these custom types and parsers in the rust output file is determined by depth. First search. We generate a custom type and part. We generate a custom type and parser whenever we reach a leaf node and then generate the parser combinator for the parent nodes, when there are no more leaf nodes found for that parent.

B

So, let's walk through parsing a tcp header. The overall protocol data unit is a tcp header, which is a struct type in our custom network packet representation system. So this is the root of the depth. First search tree and we'll generate the parser combinator for this last after we've generated all the required dependent, parsers and parser combinators, so the first parser will be for source port. This is a 16 long, 16 bit long bet string and was the parser that we walked through earlier bet.

B

Strings are leaf nodes, so we move on to the next child of the root tcp header node destination port. This is also a bit string and therefore a leaf node. So we write a custom type in a 16-bit parser. For this, the first non-bit string we encounter in tcp header is options which is an array type.

B

The elements which could be present in the options array are tcp options. Tcp options is an enum type with a limited range of possible choices. Each of these enum variants are described in their own small ascii diagrams. In another section of the same document.

B

This makes each enum variant a struct type in our network packet representation typing system. In this case, eol option is a struct. The value of the field in this ascii diagram is a bit string. This means we've finally reached the leaf node and we can write a custom, rust type definition and a non-parser for eol option kind and then a rust type definition, errors, type definition and a parser for its parent node e eol option.

B

We find that there are more tcp option variants, so we repeat this process for each one. Once we have written parsers for all of the variants, we can write the rust type date. We can write the rust type definition and parser combinator for the parent nodes, tcp option and then options.

B

The last field in this packet is the payload which we can process relatively easily as a vet string. Finally, we write the verse type definition for the tcp header struct and the parser combinator to process the entire protocol data unit in one function call we also create a context: option object which all cursor functions have access to.

B

So to recap, the system that was developed in this project. We have the machine, readable protocol document in stage one with our minimal changes to ascii diagrams and text descriptions. We have the custom protocol typing system developed in stage 2 our network packet representation language in stage 3. We have the results of my internship work. A rust library file containing correct usable parser functions automatically generated from the information we have in stage two.

B

Remember earlier, when I mentioned that, I think of these basic types and parsers as building blocks to go further. With that analogy, a tcp header is a lot like this lego block dinosaur. Here it contains many different, complex elements and it's difficult to build manually without making mistakes.

B

Our generated, parser libraries are not only a manual explaining how this data should be parsed. They also allow protocol developers to build the struct with extracted values with a single function call. This is ideal for protocol testing.

B

The picture on the left is a genuine sample of our generated tcp parser code from our modified tcp rfc document. So conclusions.

B

Initially, I decided on rust as our first parser output language, because um I enjoyed using rust for systems programming on a previous project using partial, combinators and norm turned out to be an ideal fit for parsing protocol data since parser combinators and assigning types to network protocols both use depth. First search parsers can be difficult to write manually and are prone to containing errors automatically generating partners minimizes the chance chances of some of these errors occurring. For example, the number of bets being read will always match the specification.

B

The typing guarantees offered by rust helps us to be sure that we're using the correct types in both the machine, readable protocol specification document and in our network packet representation system. If there are errors, the rust compiler will alert us to this next steps.

B

This project is still ongoing and there are more directions that this research could go in.

B

We're aiming to introduce our system to the ietf to see how this system works in practice with helping with protocol development, we need to put in more work on function types so that we can create things like encryption and decryption functions for protocols like quick, which heavily rely on this we'd, also like to use these generated thrust. Parser libraries for automated protocol testing and error correction and to support more output languages in the future.

B

Resources for this project can be found at these links. We have a peer-reviewed publication, which goes into more detail about our network packet representation, typing system and the github repository containing the code for our automatic rust, parser generator.

B

Thank you for your time and I'd be happy to answer any questions.

B

That was brilliant.

B

Thank you so much.

C

Thank you. So I know we have like a 25 to 30 second delay to um the stream, so uh just to get ahead of time. I have uh two questions. You don't mind. So the first one is there's a push for uh native uh implementations of the networking type. So the rust, like standard library, doesn't use lip c anymore, but directly operates with system calls.

C

Do you think that will affect you in any way or like developing new types.

B

uh Potentially I mean we uh so the whole point of us developing the network packet representation system was to have something that was completely agnostic of any programming languages or output. Libraries we'd want to use in the actual parser files themselves, so it should be.

C

B

For us to adapt to these things, I would think cool like I think um we could maybe sort of we might have to consider like how would we convert from network packet representation to different uh codes featured in the uh different types sorry featured in the output code files, um but that's relatively straightforward. I think.

C

Wonderful, uh so this feeds into my other question. So I guess you can use um the higher level parsers for let's say: tcp udp. What not, uh regardless of the underlying types of like uh ipv4 versus version, six.

B

Yes, uh so what we're aiming to do is have these run through um a single protocol specified in a draft? You know it's very rare, that you would have an rfc that specifies multiple protocols. um So if you.

C

B

To make an ipv6 generator sure go ahead, run it on that rfc. So what we're aiming to do is introduce our machine, readable, ascii format to future ietf drafts, and hopefully we'll see more adoption of that. So we can have automated testing going forward, but what we've done for showing the tcp example we've gone through an old rfc and made minimal changes to it to allow us to generate passers.

B

So if you want to do that with all protocols, yep, that's totally fine as well cool. So I guess an answer to your question. um Sorry. The question was about multiple protocols: sort of nested yeah.

C

If you can use the departure that's coming out of the rfc for tcp, you can use that for v4 and v6 and netlink, and what the illustrator is.

B

uh Yeah yeah: we can use this for all sorts of different protocols. I think the nice thing about parser combinators is: if you want to get really ambitious and say: okay, we've got an ipv6 packet containing a tcp packet. You know you can have a personal computer. That does that, if you like, oh.

C

Yeah, yeah cool, wonderful, there's, also a question from the audience: uh how do you deal with uh non-bite aligned structures? So, if, like a five bit, word crosses the uh eight bit alignment.

B

uh So we had so, I think I had a small file for testing when I was doing the internship about okay, what if this happens and non-bite aligned? Words was one of them, so what we found was with the bit level parsers. It tends to just go straight into the next uh bite. If you happen to count to exceed seven, um so it will just run forwards. You know, quite happily, we haven't had any issues with that. So far, norma's been very good to us.

C

B

C

It has been released right version. Six has been out since uh tuesday. I think.

B

uh Yeah, so I don't surprisingly, haven't attempted to update that yet uh so I was just writing the number five, but.

C

B

um We'll see if it still works with six and if there's anything that we need.

C

Yeah wonderful! uh If this were a physical conference, I will probably meet jeffrey who wrote the thing sure.

B

Yeah, I would love to.

C

Wonderful, uh do you want to uh precise something or say hey? This is like something that came to mind just now,.

B

uh No, I think I've said everything I want to say in the presentation, mostly uh so what we've? It's mostly a proof of concept at the moment, so I've posted the link to uh the repository and our paper explaining our system in the uh conference room chat so.

A

B

Want to take a look at our library have a play about with it see how the generated dress codes looks. um We would happily take feedback if people want to help us improve our powers, um so we consider myself still quite a novice at rust, uh so we wrote everything using uh non-functions as opposed to the macros, so we could sort of expose how much was going on just so. We can make it clear to ourselves.

B

So if people want to talk to us about how we can optimize that or make it a bit cleaner, more improvements that would be great we'd love that cool.

C

You wonder well, wonderful, so, to the lovely people in the stream. This is about the last chance you get to ask more questions. uh Hs vision has one: has the itf been receptive to the machine, readable diagram format.

B

uh So the problem with the itf is there's so many different groups, it's impossible to get like a group of consensus for the whole organization. So what we've got at the moment is a small side meeting uh the formal descriptions and techniques side group I think which is aiming to sort of like say: okay, how can we deploy this, but we have um so stephen mcquiston and colin perkins.

B

uh Two people involved in this project are heavily involved with the ietf, uh so I think they're having discussions to see how we can get this um more widely deployed. So there's been past attempts about like okay. We can have custom tooling to do this this and this and it's all singing all dancing, but what we tried to do was make something relatively simple and unintrusive that could work with multiple workflows.

B

um So that's why we're hoping this will see wider adoption cool, so the.

C

B

You haven't published using it yet but watch this space.

C

I guess you will be trying to investigate like the correctness of uh the middle boxes and whatnot or.

B

C

Try to circumvent them.

B

Cool yeah, so one of the examples that we're working on at the moment is quick with quite being very high profile and a very complex protocol. So we think if we can successfully parse this- and you know um we can successfully use it for testing, then we think that's quite a good promotion. I suppose.

C

Oh definitely yeah I mean I mean having an actually correct implementation. That's like done when the specification is finished. Well,.

B

This is the only problem. This was one of the main motivations about. Like you know, you get protocols that are becoming increasingly more complex. Like quick, it's not surprising that there will be stuff, uh there will be flaws in it. So.

C

B

We got a packet generated by c and c based implementation of quick and we fed it through our rust power. So we could, you know, potentially find errors that way so.

C

It's interoperable.

B

With uh stacks written in other languages, we just need to serve output that they generate.

C

B

C

Tools like cargo expand. I could then see at the generated code and maybe check out the state machine that has been generated to see if, like the specified, uh behavior makes any sense right or if there's like obvious flaws in the standard that you cannot see by just reading the text.

B

Yeah um yeah: that's the aim to sort of catch the subtle bugs where it's like. Okay, you know, um essentially what our partners are testing is your output on the wire correct. You know, is it doing what you think it's doing? We can maybe come up with surf like more advanced testing, harnesses and sort of automated error correction things for stacks later on, possibly, but that's going to take some time to develop.

C

B

C

Looks like a long ongoing project with very.

B

Short potential.

C

Hopefully, yeah wonderful, so uh I'm currently not seeing any more questions. I hope I haven't missed any.

B

No, it seems like that's all of them.

C

Wonderful then, uh thank you again very much. Thank.

A

You for having me.

C

C

C