Rust Programming Language RustFest Global 2020, 23 Dec 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: tkat0 - エッジとクラウドでRustを使いこなす～AI/IoTでの事例～ — RustFest Global 2020

Description

English title: Full use of Rust on edge and cloud: AI and IoT use cases

この発表では、私たちのプロダクトの開発事例を通して、エッジデバイスやクラウド側でのRustを使った開発をする上で、実践的な設計実装のテクニックを紹介します。

More at https://rustfest.global/session/10-%E3%82%A8%E3%83%83%E3%82%B8%E3%81%A8%E3%82%AF%E3%83%A9%E3%82%A6%E3%83%89%E3%81%A7rust%E3%82%92%E4%BD%BF%E3%81%84%E3%81%93%E3%81%AA%E3%81%99-%EF%BD%9Eai-iot%E3%81%A7%E3%81%AE%E4%BA%8B%E4%BE%8B%EF%BD%9E/

A

This presentation is the ‘full use of rust on edge and cloud: ai and iot use cases’ by tkato.

A

Hello., I’m tomohiro, kato.

A

I work online with the id, tkato.

A

I’M a software engineer at the japanese company, mobility technologies.

A

At my job, I use rust to develop a machine learning system with an edge device.

A

I hope you enjoy this talk.

A

In this presentation, I will go through product development examples and introduce design techniques.

A

Developed with rust on an edge device or cloud.

A

This is my agenda.

A

I’ll first introduce drive, chart, our product, and then developing rust on edge and cloud.

A

First is an introduction to our product.

A

We are developing a product called drive, chart.

A

Ai drive chart is a service to help reduce traffic accidents using an ai for taxis and commercial vehicles.

A

It uses the drive recorder and cloud to move the ai algorithm in relation to camera image and sensor data.

A

And creates a safe driving report.

A

I’Ll introduce the part of drive chart developed with rust.

A

This is computer vision, processing with production on edge and an evaluation simulation with development on cloud.

A

We implement a computer vision algorithm with real-time, deep learning on the drive recorder using edge.

A

We slightly alter the edge implementation on cloud and operate it on kubernetes.

A

Next, I’ll explain how this development progressed by adopting rust.

A

We initially launched using c++., we considered switching to rust due to the development time and quality issues.

A

However,, I’m not saying that c++ has issues., just in our case with limited time and manpower to develop.

A

Rust was more suitable.

A

The first issue is the build system and test framework.

A

These aren’t built into c++,, so we had to select our own technology and write most of the script for it.

A

We wanted to simplify the non-code script, to focus on substantial system design and coding.

A

Rust prepares the build,, test, and documentation from the project template via cargo.

A

That solved this issue.

A

The second is safety..

A

The drive recorder requires high-quality software, so it won’t crash, even if there are issues during operation.

A

Rust mostly removed the the memory operation bugs that we had during the compilation time.

A

Since we switched to rust, no bugs causing it to crash have been imbedded even during its development..

A

Even rust, can’t prevent the logic bug, so be careful.

A

Because it had these advantages and we expected the same execution, speed and memory as on c++.

A

We actively switched to rust.

A

In about two weeks,, we created a prototype to make this switch.

A

The core was essentially the same as on c++., we implemented the application on rust.

A

And checked the drive recorder’s operational performance.

A

We didn’t rewrite everything all at once.. We slowly switched what worked from c++ to rust.

A

Only then did we decide to develop a product on rust.

A

Afterward, we progressed while improving our knowledge of rust and launched without any issues.

A

Part of this is still on c++,, but it was mostly implemented on rust.

A

Next, I’ll introduce the development flow.

A

We develop the ai algorithm on python, reimplement it on rust, and operate it in the production. Environment.

A

Finally,, an ai function was supplied as an ai library and incorporated into the drive recorder, application.

A

This is an example of how we add a new function.

A

First, we do r&d on python and verify the algorithm.

A

This is because python is the main ecosystem to develop deep learning.

A

Next, we verify this while integrating the python module and edge ai library.

A

Once the deep learning module is finished,, we can verify it by adding it on the drive, recorder’s production, pipeline.

A

In this development on cloud,, we use massive test data, including videos and sensor data.,.

A

We evaluate the overall algorithm and machine learning system.

A

Next, we reimplement and verify python on the ai library, with rust.

A

Finally, we incorporate it into the drive. Recorder, repeatedly verify the recording, and go live to release it.

A

We can efficiently develop new functions by seamlessly linking r&d with production.

A

We brainstorm software designs to facilitate this.

A

Let’S get into the design and implementation.

A

First, I’ll explain how to handle deep learning and the computer vision, processing.

A

On the drive recorder with rust.

A

We write using a standard library on rust and an image processing, crate.

A

We wrap the c++ library with rust as needed for performance.

A

We use our successful c++ library for deep learning processing,, but I won’t go into too much detail.

A

We create our own api on c for the c++ library and wrap it with bindgen.

A

But we use cpp when we want to quickly call c++ from rust.

A

For reference,, we shared the information on the rust machine learning, ecosystem.

A

There is already rust binding for major frameworks like tensorflow or pitorch.

A

Also,, this runs on the hardware of the common rasberry, pi and smartphone,.

A

So tensorflow and tvm can be used on rust.

A

In addition to this,, I recommend binding the c or c++ library on rust, like we did.

A

Let’S go onto the unit test.

A

Rust is particularly reliant on third-party crates, and they sometimes begin to degrade because of this.

A

I’Ll introduce several measures to prevent this.

A

Firstly,, I recommend a snapshot, testing method using a crate called intsa.

A

This crate stores the execution results of the test and can easily compare results while running the test.

A

We use this to check that the system detection for videos is consistent.

A

The automatic test runs on github actions and includes snapshot testing.

A

This is useful, because there are many useable commands on github actions in actions. Rs.

A

The dependency version recorded on cargo.toml goes up to the patched version.

A

The github dependabot will take care of updates.

A

The dependabot creates a pull request for any crate, updates, automatically test it, and verify if it’s degraded.

A

This makes operation a breeze.

A

Let’S continue onto development on cloud.

A

We are constructing a simulator to evaluate edge implementation on cloud.

A

As shown in the diagram,, we join multiple crates in the application on edge as a monolithic, library.

A

On cloud,, we deploy crates as containers and structure it as a distribution system.

A

There are two reasons for this..

A

The first is it accelerates, building.

A

It splits containers to hasten updates and deployment per function, with differing development, cycles.

A

The second is it integrates r&d and production.

A

In one use case,, we wanted to evaluate the deep learning module.

A

Within the current pipeline with python implementation during r&d.

A

In brief,, we selected an architecture to create a mixed system with rust and python.

A

You can consider other methods to wrap rust as a python library and structure. The overall system on python.

A

We selected this structure for its loosely-coupled,, simple structure.,.

A

We collaborate on shared code for edge and cloud within the crate.

A

And give it a design with hexagonal architecture to continue developing it on edge and cloud.

A

Edge and cloud have a common core logic.

A

But the implementation selection depends on the application, entry point,, the hardware, and the use case.

A

For example, during edge production,, we incorporate rust implementation in the application as c api.

A

When we set a device benchmark from a pc or call from other applications on kubernetes,.

A

We use the grpc or cli interface.

A

The inference function allows us to select implementation to call the r&d python model with grpc,.

A

A deep learning framework for the device, and a mock test framework for x86.

A

This seamlessly links from the r&d on python to the production development on rust.

A

Switching over functions is implemented with rust traits as needed for each use. Case.

A

This slide shows implementation, examples.

A

The code is abridged but sufficient to get the gist.

A

This allows the module inference to be switched to three different types: grpc,, edge, or mock..

A

First, it implements the trait module service.

A

Consider the input, image and inference results for the execute method.

A

Next, we implement each trait.

A

The gprc module service uses the grpc client, like tonic to access the external api.

A

The code for the edge module is abridged.. This is implemented through a deep learning framework for edge.

A

The mock module service is a unit test implementation that responds when there’s an instance.

A

This is how you create multiple implementations from one trait.

A

I’Ll explain the part of the code used in the application.

A

This helps to switch the implementation when starting the application.

A

This uses the trait object to reset different implementations based on the given module type variation.

A

You just input model.execute to call it.

A

This abstracts the implementation and lets it switch depending on the use case.

A

We used these techniques to facilitate an improved testability and integration with systems other than rust.

A

Our final technical topic is logging.

A

We often use log to debug and profile at the application. Level.

A

Naturally, we use a benchmark like cargo bench during development at the crate, level.

A

We often use json structure logging,, then parse and visualize it with a tool like chrome, tracing.,.

A

This lets you see, processing with multiple processes and threads.

A

This is a reasonable method to get an initial indication of bottlenecks.

A

The log base doesn’t rely on language as it’s on a system constructed with multiple languages, like rust and python..

A

You use a s-log crate on rust to do structured, logging.

A

You can output a json log with a macro, like in the diagram.

A

This shows each inference begin and end event in the log.

A

Let’S discuss system changes within half a year of development.,.

A

In a few words,, the system changed from a monolithic crate to a micro, crate.

A

In our initial development with rust,, we often stuffed multiple functions into a crate,, but overtime split. Those up.

A

There are several reasons for this..

A

First, is the build time. differentiating crates accelerates this, since we don’t build all functions at once. During development.-.

A

One advantage is how easily it combines with other languages once you can make a container per function and crate.

A

In our use, case,, we wanted to make a system that mixed python and rust.

A

One benefit of employing a microservice is how flexibly it selects technology per function.

A

I recommend the option that doesn’t force you to use rust for several lines, written on a shell script.

A

Our goal for this microservice was to improve maintenance.

A

Currently, our team has 4 people, writing rust, and is a small part of the overall company.

A

We value making it simple so that any new participating members can quickly catch up.

A

Lastly, I’ll give my impression of using rust in product development for half a year.

A

I felt rust was a good choice,, because its team development is good and production would develop. Quickly.

A

Since I proposed switching to rust,, the team has grown to its current size of 4 people.

A

Rust was the first programming language, any of us experienced.

A

Familiarity is the key. we got used to rust through the compile error messages and the clippy messages.

A

Earlier I spoke about system transitions.. We were able to work on stress-free incremental development.

A

Because, even when we conducted major refactoring to differentiate crates,, we could check for slipups, while compiling.

A

I spoke about our initial motivation to use rust.. I think it has an enhanced ecosystem.

A

In product development,, our team focused on the design and coding without overwhelming the repository.

A

With script on build and test documentation.

A

Another positive is that the role of our team expanded.

A

Rust can be developed on a variety of platforms and domains.. We mainly worked on the development on devices.

A

But we could sufficiently develop on cloud too.

A

To conclude, today, I went through product development examples and how we used rust on edge and cloud.

A

We devised a software architecture on edge and cloud to integrate r&d on python.

A

To the production on rust.

A

This approach and implementation were based on our use case and may not be applicable to everyone.

A

Still I’m, please so many joined this session.

A

Let’S all try out product development on rust.

A

That’S it. thank you for listening.