Rust Programming Language Oxidize Global 2020, 22 Sep 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Exploring Embedded Rust for Functional Safety

Description

Jack Greenbaum from Green Hills Software shares his experience with exploring Embedded Rust for use in functional safety systems at Green Hills Software. This includes the parts that are easy, a few rough edges for the end to end Embedded workflow, and recommendations for bringing Rust to the broader Embedded Systems market and Function Safety Systems in particular.

Interested in using Rust for safety critical systems? Reach out to us at sealed-rust@ferrous-systems.com!

A

Thanks for the handoff james, all right, I'm gonna go.

B

Ahead, go ahead, you go. You go ahead. Okay,.

A

I just want to start off by thanking everyone for for being here at these odd hours and odd places. This is definitely the type of event that I would have uh traveled for and and really enjoyed the conversation so I'll be watching the chat and uh well, let's see how this works.

A

So I'm going to start tonight with an introduction to green hill software and functional safety. Functional safety is one of those things that it sounds like it's something you ought to know what it is. But then, when you dig into it, you don't so it's sort of a cryptic topic for a lot of folk I'll talk about why uh you might want to use rust in a functional safety system and and explore whether it's suitable for functional safety systems. Today.

A

So here's a an overview of green hill software, we're the largest independent provider of software for embedded systems. We've been in the industry for 35 years.

A

We started out as a compiler company in the late 20th century we added an integrated development environment and then a real-time operating system to our offering, and today we're the leader in safety, qualified tool chains with our our multi-ide and green hills, compiler and safety certified operating systems with our integrity, real-time operating system, I've got these uh posters, uh we use these at trade shows we have them around the office and they they highlight uh different customers, and this grouping here shows our traditional market of safety systems, planes, trains and automobiles.

A

If you drive a japanese us or european produced automobile, then you likely are have some green hill software in there, because we've been shipped in 130 million vehicles to date in aviation. If you, uh when, back when we used to fly on commercial aircraft, you probably flew on an aircraft that used green hill software.

A

All modern fighters in the west use integrity for their operating environment and a lot of transportation infrastructure trains is built around our integrity rtos, but safety markets are much broader really. They also include things like agricultural equipment.

A

um These have whirling blades of death and when you want the the whirling things to not be whirling so it's safe, then uh you know you want assurance that that it's going to be safe when you know, especially when the operator is out around the equipment, so there's safety, certification standards for agricultural equipment.

A

Medical is interesting because it not only has safety but also cyber security standards these days and that that's a new requirement and then there are kind of niche markets. um Nasa and jpl use us in space flight equipment as well as in defense and then yeah. That's an hp printer.

A

You know the cost of a failure in a consumer appliance, isn't a safety issue, but it's a big business issue, so reliability matters there too.

A

So what is functional safety certification and how? What role does it play in the in these products?

A

Well, functional safety certification provides assurance that a system was built taking the safety of the system into account now notice that wording, it doesn't assure that the system is safe. Only the processes that are essential to building a safe system are followed, see it's a little cryptic.

A

It's assurance that you followed a safe process, because that's something that you can measure examples of functional safety standards include the usfaa do178c for software used in commercial aviation, iso 26262 for automobiles and software in automotive, uh electrical systems and iec 61508 for industrial automation, factory, floor equipment.

A

Now the interesting thing is even though we've got these different standards from different fields, they're, all pretty much the same, only a little bit different.

A

They all talk about goals, supporting processes and work products, and this is really the secret decoder ring. If you ever go to read a standard like 26262 and you're wondering well, what do I have to do so that my product is certified?

A

What you're looking for are the processes.

A

So the processes are things you do to achieve goals and work. Products are what you show an assessor to prove that you followed the process.

A

It sounds simple: it is that's a lot of work and there's a lot of details.

A

The good news is that the processes and work products are are typically tailored to the risk level, for example using different types of structured methods for risk analysis based on risk impact. If the product is low risk, you don't have to look too hard.

A

All of these standards have different levels of assurance where the highest level is for life, critical systems and the lowest level are for systems that have much less impact. For example, an instrument cluster in an automobile, a safe state for a graphical instrument. Cluster is actually blank.

A

The safety rules. There say it's better to show no information to the driver than to show incorrect information to the driver.

A

Now that is considered a a middle low safety level now compare that with uh an ecu controlling a hydraulic system on a commercial aircraft.

A

If that fails, it could be catastrophic, so that requires a higher level of assurance and therefore more work, more processes, more work products to achieve the higher goals. So that's what functional safety certification is is really all about.

A

So typical development of safety systems follows a v model of development down the left side, you do the design work to avoid systematic faults that are, that is faults that are due to failures in your systems, failure and your processes to engineer a correct system.

A

You should be able to anticipate these and mitigate those risks.

A

Now, on the upper side, heading up the other side of our v model, we're going to contain the faults that are unavoidable.

A

You know if the if the chip fails, the chip fails. What is it that you do about it? What is the system supposed to do when the chip fails?

A

So that's what you provide assurance for on uh on the upward side of the v model and at each level of abstraction you pair a design process with an assurance process.

A

So you assure your detailed design through unit testing, you assure your high level design through integration, testing and requirements coverage through system testing.

A

So this is the typical flow of development in a safety.

A

System: here's why we are interested in rust. We have a theory. We have a theory that using rust, reduces systematic faults due to memory related, bugs remember that 70 of bugs that james talked about in the keynote.

A

So we see the rust borrow checker enforcing correct by construction, pointer use and memory allocation.

A

This is a big deal because that's where the bugs are, and not only is that where they are they're there, because they're very difficult to test for, because they're very state dependent.

A

We also think async await, is a big win for rtos application code, readability and therefore reliability.

A

You know today for event driven code. We often are writing state machines with big switch statements.

A

Async await unwraps that state machine and makes it read like procedural code, and humans are much better at procedural code than declarative code.

A

That's why we're not all programming in reactive, functional programming, languages at least not commercially, and then we think that um the benefits of rust actually extend above the unit level.

A

Correct composition is also enabled by the borrow checker, so we think it reaches further up.

A

But there's a context here.

A

You know that context is this v model and it's been developed over a very long time. You know this. Isn't functional safety, isn't a new concept, the v model, preventing systematic faults. It's it's not a new thing.

A

So you know when we bring in rust.

A

Rust has its it's a c foreign function interface and we think that's how rust has to be integrated into safety systems for the foreseeable future.

A

You've got rust, maybe you have um other compiled code from other authoring systems like uml or other model driven tools like from system design. Companies like mathworks, but in the end, it's a sea world.

A

All of the processes that get you to implementation and then back up are geared around a c compatible abi.

A

Now the world of embedded systems is a bit different than general purpose operating systems and this importance of a c tool chain.

A

Really goes very deep into what is different about embedded systems.

A

The big thing is: embedded systems are diverse,.

A

There are diversities in the there's diversity in the type of cores that are used. You know arm is, of course, very, very, very, very popular and embedded these days intel's. There also there's a lot of vehicles driving around with arm and intel chips, especially in the instrument clusters.

A

You might be surprised to know, though, that there's a lot of life left in power, pc power, pc, is still we're still seeing new starts in automotive and also in aviation, both commercial and defense.

A

These chips are really well known, they're worst case uh performance um and there's a lot of code code lasts forever code lasts a long time.

A

So when you have an operating system that you've been flying for a long time, you don't want to change it because that's risk people stick with power pc because to them it's a risk to move.

A

Now, a newcomers risk five we're seeing some applications that are on power: pc move to risk five, instead of arm for future proofing, it's seen as got having a longer tail more vendors, avoid vendor lock-in and then there's kind of some funny processors that you mostly only see in automotive like the infineon tri core and the renaissance rh-850.

A

So why do you care that? There's all these different cores? Well, it's because they all come with different tool chains.

A

At green hills, our tool chain supports all of these processors and that's nice for the customer, because the compiler flags, the linker everything is, is all the same.

A

But you know arm has an arm compiler. They want their customers to get the most out of their processor core, so they provide their own compiler intel does the same.

A

Then there are independent vendors like iar, and then we have our open source alternatives. Gcc clang.

A

So you know, there's diversity here and where it really hits home is in the linker.

A

Linkers are actually pretty magical things. They perform magic in embedded systems.

A

In the general purpose os you almost never interact with the linker. Maybe you need to set an ld library path or make sure a dll got into the registry or something, but in embedded systems. They are touching your code in a very first-class way.

A

For example, linkers will modify your code to reduce size. Things like far call patching some common sub expression refactoring in the binary code to reduce code size and then we've got overlays that'll take parts of your code chop. It up put it into separate binaries.

A

So you know these are pretty pretty deep features.

A

Now. They'll also modify your code for functionality, and this typically happens with silicon errata workarounds, where you throw a magic flag and all of a sudden, a no-op is inserted in between two instructions where it hadn't been when it came out of the compiler.

A

This is a real, safe way to patch errata that are exercised by certain instruction sequences because it doesn't matter where the binary came from as long as it's linked by this linker.

A

And linkers also feed downstream tools. You know they tell the bootloader. What's live, what's not what needs to be cleared and where to load things?

A

So linkers are first class objects in an embedded chain, an embedded tool chain, whereas you just don't see them much uh in general in the general purpose world.

A

Now these magical features they're often implemented as bespoke features.

A

So this entrenched c based ecosystem really goes all the way down to the linker and if you're going to bring a new language in, it has to support the diversity of cores and the diversity of linkers diversity of tool chains throughout the ecosystem.

A

An entire rust build is is a ways off because of how long code lives and not just the code, but the supporting processes and infrastructure around your application. You know take a look at build tools. We have a builder, we like it. I like it.

A

A lot of our customers use, make files, they use, cmake or cons or bitbake or cargo.tamel.

A

There's a gajillion build tools and if you want to fit in, you got to fit in with the build tools and the operating systems and the test tools. You know bespoke test runners, that's what you find in embedded systems.

A

I I didn't have time to get a picture of the the test lab at green hills. Our test labs. We have a board for every bsp that we support and connected to. It is a jtag probe, a serial port, a network power supply and an ethernet port, and we have hundreds of these and we're running validations constantly, and you can't do that with an off-the-shelf system.

A

You have to tailor it to your needs and that's what our customers do as well, and there are other essential tools in the build chain that are used to this cbase flow. Things like license.

B

A

That read your code and look for whether you, whether your engineers accidentally cut and pasted from open source and look for open source, license headers, and then things like secure boot signers, you know. So if, if rust is to be adopted it it can't be tied to a single tool chain and embedded, and this is really what's what's different about embedded and therefore functional safety.

A

So where are we today.

A

So if you go look at the rust embedded book, you'll see a system that kind of looks like this: it takes rust and a cargo.tommel as input and it'll connect to a cortex m3 based system. It suit the nuts. You know it frank. This is a really good system.

A

If you haven't done embedded systems programming before and are curious, this is a great system. There's a talk later on in the conference about is rust. Embedded the new arduino well take a look at this board over here this board's literally sitting on my desk. This is a an st nucleo board.

A

They provide arduino images for this board, the daughter card. That's an arduino, compatible nxp sensor module. I bought it for a geek time project that that I'm doing.

A

The cortex m3 based software hardware, abstraction layer that rust embedded provides the rtfm real time for the masses. This is a great system for building small embedded systems and it just works, and that's really one of the things I like about it, and that's why I really do believe yeah. This is a better way to go than an arduino.

A

So, let's just look one level deep here: um what's different about programming, rust embedded, then uh any other rust platform. Well, one thing is this: is this link map this link.x file? This is input to the linker. Remember I was talking about linkers, so in embedded systems, you have to tell the linker where to put things, because you've got a lot of different types of memory and embedded systems that you don't have in a general purpose os and the general purpose os everything's, just memory, it's just ram and pages, it looks infinite to you.

A

Well memory is not that, like that in embedded systems, you have flash typically you're booting from flash rust, embed it'll flash it for you. It's really awesome, then you've got to manage your ram and sometimes you'll have different types of ram fast ram, external slow ram, so that linker command file is is really important. So that's a that's a difference here and then you've got the cargo run command that doesn't just shell out it actually talks to this part of the board here is: what's called this is a small jtag run control thing?

A

They call it st link and cargo run for cargo embedded, actually talks to this thing and resets the whole board and flashes memory images through this and provides your debug interface.

A

So really it's providing a link map and cargo run. Okay- and uh you know that's, what's different here, oh and a bunch of really awesome crates.

A

So what if I wanted to use this system with something other than this um arduino looking board.

A

What if I wanted to use it with something more industrial strength? What what do I have to do? Can I take rust embedded as it exists today and use it in a more commercial manner, so this board I'm showing now this one's on my desk as well, while actually on a little rack next to my desk- and you know, some things to note is compared to the hobbyist nucleo arduino baseboard. This one's got a display. Okay, it costs a little bit of money. It's got canon rs232 down here, but really importantly, it's got an arm.

A

Cs20 trace connector and I've got it connected to a green hills trace pod.

A

If you've not ever used trace debugging, oh man, it can be so useful, especially in embedded systems where, because you don't have memory protection, often when bad things happen, really bad things happen and you'll end up in an error handler and you'll often wonder well. How did I get here?

A

This trace pod connected to the trace port with the appropriate electrical on this higher cost board. Lets me do that, so the extra peripherals, the extra debug? um That's what makes this a more commercial oriented board? This is, in fact, a reference board. That's frequently used for medical designs, infusion pumps and those types of systems.

A

So I asked the question: what do I have to change if I want to use this board? Well, the first thing is: I already have this board up and running with the green hills tool chain. I've got a green hills, linker command file, oh shoot, but I'm using the rust linker here. Okay, what am I going to do about that and then cargo run cargo run knows how to talk to open ocd devices and run control them.

A

The greenhouse probe is not an open ocd device, so I can't just immediately use cargo embedded as its shipped, but I don't have to go very far and that's because cargo will also build a static. Lib it'll build a crate as a static library, a dot, a file fully linked with everything, but the operating system, this dot a file, looks to a tool chain like any other dot a file.

A

So all those integration points are now super easy, because now I can use my ghs linker command file and c code that compiles with the green hills, compiler and even in rtos, because I can pull that static library into my green hills builder. This is our our project manager.

A

So with that static library and just changing my my crate to be a static lib instead of an executable, now I get into the green hills tool chain and now I can run with green hills. Multi and now I can do trace-based debug on this commercial board.

A

So you know the static lib integration.

A

It means we, we don't need to modify cargo to know about a different link or we could. We could modify cargo to know about a different linker.

A

We could modify cargo to know how to talk to the multi-debugger, but but we don't have to do that and we don't have to modify cargo to run test tools. Secure boot, encrypters license scanners.

A

This is a really good integration point and it's supported today. Now there are downsides, we lose cargo run. I put that with strikethrough because I'm not sure in an embedded world. I'm not sure cargo runs a great thing for the use model of an arduino yeah in a commercial world, but cargo test is a loss.

A

You know having an ubiquitous test runner and really um well supporting test driven development. That's a really good thing, but you know cargo test today isn't exactly embedded friendly.

A

So I want to use this opportunity to invite the community to consider how cargo tests can be targeted to embedded.

A

You know like right now: cargo test without changes- or you know pulling in things- uh requires a test runner. You need to be able to run, and that requires a lot of integration. If you're going to pull it into a test environment like we have at green hills and our customers have.

A

But what, if it just output a static lib, then we could integrate, but then we get into things like the size of the binary. I don't know if features are enough to where you could just keep rebuilding and get static libs for each of your tests to make sure that they fit in the amount of ram.

A

I love cargo test when I'm running on a general purpose, os I'd love for it to be more embedded.

A

A

Okay, so you know for general purpose: embedded non-safety, critical development, I've been able to use rust embedded, and I really like it- I like it so what's really missing for um for safety using it in a safety system.

A

Well, safety, qualified tool chains is is a gap, and if you saw um james's uh keynote leading into this, then you know a bit about sealed rust and we, like sealed rust. We endorse sealed rust.

A

You know you really need a language standard, that's on the sophistication of iso c, to support qualification requirements.

A

Now, if you look at the backing material behind sealed rust, you'll see that james calls out and ferrous systems calls out, and the supporters of sealed rust call out that rust is already in a really good place because of the test infrastructure and the rfc process. You know the rationale for the design choices is there. This is a good place to start from, but the level of rigor needs to be brought up the level of formality and stability.

A

um You know, I've said a couple times that code lives forever code lives much longer than you ever possibly imagined. Stacks are really an example. Some of the can stacks in automobiles from the big companies 15 20 years old same software. They just keep rebuilding it. It's too expensive to test again.

A

So the stability in the language that sealed rust you know, takes beyond the quarterly releases that provides a level of certainty that the long-lived code and safety systems needs, in addition to the formality of the language, so that accurate testing and test coverage can be achieved.

A

So that's a piece: that's missing for safety, there's some smaller things too. In the c foreign function interface a couple niggling details like c volatiles and c bit fields.

A

You know it's a sea world today and it will be for the foreseeable future and I know there's spirited discussion about volatiles and bitfields, um but I think these need to be not just crates, but either standard crates are more than just a pointer type.

A

um I think there's more to do here and then another aspect of the c4 and function interfaces is, you know, bind gen. Is it exists? You know binden open source buying. Gen, I think, aspires to the wonderful ideal. Wouldn't it be great if you could just call c from rust and just hit a button?

A

It's it's not that simple.

A

um You know binding's built with klang's front end and if you have a different set of headers well, it's not quite gonna work or if clang is a little bit different than the c compiler that the code you're trying to translate uses- and you know as advances as our standards are these days, there's still always little differences in c compilers, so you know, bind gen, just isn't, is portable or and and it has some attitude about volatiles and bit fields.

A

I don't know it's, it's not complete and I don't know if it ever will be, and it probably doesn't even need to be.

A

But a conclusion I've come to is that each tool chain will need its own bind gen to really help ramp up the.

B

B

A

Well, that's a nice internet glitch, so in summary, russ today is ready for use in non-safety critical, embedded systems.

A

We think that static lib is the best integration point today, there's still some work to do with the c4 and function interface level, but I mean this is a usable system for building in non-safety critical, embedded systems to take it over the finish line and really get the advantage of the language that we all see.

A

We think that sealed rust or something like it is needed to enable use and safety systems and and that's what will unlock commercially available qualified tool chains. I don't think any one vendor or any one program adopting rust is going to be able to pay for all the process that provides the assurance that we've built a system that properly takes the safety into account.

A

So that's the end of my talk. um James, do you want to help me moderate uh questions, or how should we take questions.

A

I think you're, muted james.

B

I am muted thank you yeah, so we've had a couple questions in the chat, so we're a little over time, but we've got time for a couple questions before we go back into break. um So the first question was from evan richter and evan richter asked jack. I would love to hear your thoughts on efforts to help the difficulty of testing on embedded platforms, fuzzing and coverage. Guiding fuzzing in particular, has really taken off in the last five years, but still it is very difficult to harness, embedded solutions into a fuzzing target.

B

Any thoughts on fuzzing or other dynamic analysis techniques.

A

Yeah, you know, one of the difficulties of embedded systems is the lack of resource, and uh so that's one problem with with test generation techniques like fuzzing, um you know we we do fuzzing uh at the api level, so it's certainly a great way to check for robustness.

A

In general, we found the usability of test generation tools to be lacking.

A

And I'm not sure how to resolve that.

A

You know we use various test generation tools, both uh in-house tools and third-party tools for test generation, they're all really awkward to use, and I think that the only way they get better is more people using them and not being willing to put up with low quality.

A

You know, I think, eda tools fall into this trap as well. The quality of electronic design, automation tools, things you would use to program fpgas is incredibly low and the cost is incredibly high, but the number of seats worldwide is low.

A

So my hope is that as more people are involved in safety, critical systems and the mandate for using safety, critical systems and functional safety, certifications is growing. Just look at the european car safety standards.

A

I think that's really the key to making these tools more usable is more people demanding better tools and, where necessary, just writing them in open source.

B

All right, I've got a second and probably last question for you um b brown in the chat is asking. I appreciated the static lib example of how to plug into an existing project, but do tool. Companies worry that their long-term market share may reduce if new projects pick a fully open source tool chain with c and c plus, plus the open source tools can be so clunky that id ees, like iar, add a ton of value, but the rust tool chain is just so easy and user friendly.

B

I'm not sure if I need an iar, although I may want the compiler back end still.

A

You know my attitude about open source is if open source is good enough for you go for it. You know um our customers find that open source doesn't have uh properties that they need for long-term support and functional safety. They don't have certifications, they don't have long-term support. They may lack features for embedded systems. Companies often pay us to customize tool chains to exactly their specifications.

A

So when open source is good enough, hey that's great. um You know, I'm not threatened by an open source, rust.

A

There's a reason that closed source exists and it will there will continue to be the need. You know, look at debuggers.

A

I think there's a reason that people on linux don't use debuggers and it's because gdb is not a good debugger.

A

It lies to you constantly about your data values and how you got there and where you are, you know a trace-based debugger like like ours or from one of our competitors like lauterbach, oh, my goodness, it just lights up the world and the information that we can show you about your system by coupling our compiler and our debugger and our linker, and give you unprecedented visibility into the timing and behavior of your system.

A

You know: that's just not stuff that open source is good at so open source is good at stuff. That's used by a whole ton of people and less good at stuff. That's used by fewer numbers of people- and uh you know we're always going to have value, is, is what I think greenhill. There will always be a need for a company like green hills, even as open source takes more and more of the software content. As you've observed.

B

All right, thank you so much jack. I know uh we'll probably have at least one more question in the chat and it'd be great to uh if, if you could comment on that on the matrix chat, um but I know you also need to get to sleep before you come back for our safety critical panel uh tomorrow for you and later today for us. So thank you so much for your talk and we're gonna go to break now and we'll be back in a couple minutes with our next talk. Thank you. Everyone.