Cloud Native Computing Foundation EnvoyCon 2020 - Virtual, 12 Nov 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Support Arm64 platform in Envoy - Lizan Zhou

Description

Support Arm64 platform in Envoy - Lizan Zhou

Envoy Arm64 will be released from 1.16.0. In this LT Lizan will discuss about the effort to make Envoy officially support Arm64 based Linux. This includes some code / test change caused by different endian / memory layout and compiler defaults, how we identified these problems, and how we built CI infrastructure.

A

Hello, everyone welcome to envoycom. Today, I'm going to give a talk about support, arm 60 platform in envoy. My name is lejan zhou. I work at tetrad, I'm also an envoy maintainer as well.

A

So first, let's go over the story of arm support in envoy. The first issue about arm support opened back in 2008.

A

At that moment we didn't have enough resource, either engineering resource or compute resource, to have the official support for arm. But there are several people in community try to build envoy on raspberry, pi or other arm board.

A

Then the open lab folks at the third party ci in q2 2019 this year. We got some support from arm and then we can. We was, we were able to add some experimental ci in q2. Then we get it officially fully tested in q3. The first official release with arm64 support was released in last week.

A

um The version 1.16. is the first version of official arm 64 image build I'm going to talk about the story, how we get the the first official release in ci and build system.

A

So, let's go over what envoy build system envoy uses a bazel as a build tool? It provides the benefit of hermetic, build and remote build cache.

A

We use a remote, build execution from google. Also, we use cache for from open source project called bazel remote with s3 back-end for arm the build is really large. Today we have just for the binary test targets. We have 744 tests.

A

Each test target have tens or 100 test cases, so this is really time consuming to run with small machines. How to make this run within time is, was also a challenge to get proper arm. Support in unvoice ci also envoy have a lot of build dependencies, namely the big one is g-perf tools which is a metallog extension and ng hdb2 or http, 2, codec and ammo and other like protocol buffers and so on.

A

Luckily, we didn't have any major issue with those dependencies uh we need. We had to do some small patches to make them uh work with arm build, um but overall that was a small part.

A

So, let's talk about the ci, the unvoice ci runs on azure pipelines. um Before we have the arm support.

A

We we run format, check and do the release and that the binary build in the release, um you'll go to the docker image and we run some sanitizers coverage and gcc gccn, etc. I omit the mac os and windows support here then, with arm. We added a new arm, 64 release job here and then we take the binaries build from xd 64 release and arm 64 release to a docker multi-arch job.

A

This one builds a multi-arch image for both um x64 and arm 64 and push that to a docker hub.

A

So there were some challenges for support arm. The first one is basil: bezel wasn't bezel didn't have official release before 3.4. um We worked with team in google to make the official um release happen. Azure pipeline started support arm 64 in q2, which is actually when we looked at it. So this was the right timing and because the azure pipeline doesn't provide the managed instance for the for its ci workers.

A

We built our own self-run agent's infra on aws and, at the same point, aws releases its new arm 64 instance graviton 2, which provides powerful instances.

A

This gives us a lot of flexibility to run the ci on large machines.

A

The ci infra basically set up idle instances in aws that waits for a job from github, and then it works on the ci job from azure pipeline. The code is in the ci info ripple and it's very simple.

A

The next is the docker image build darker now have the multi art support with the build kit, we can use same docker file to build both arm 64 and amd6 c4, which is xc60 or images.

A

We change the debug image from alpinebase to ubuntu-based to better support arm, because the alpine gtc based image that we use doesn't have the armed version.

A

So next I'm going to talk about pointing envoy to arm.

A

Envoy is a modern code base. We didn't have any major issue to build my codes into arm 64.. There are some caveats that we have to pay. Attention is the ndm and the sign chart versus unsigned chart, which is differs on compiler default for those platforms.

A

The memory size dependent test is also failed initially, because the p thread pointer size is different. This affect the hot restart version.

A

The the biggest one uh we have to handle is the exception handling initially, when we started arm building, while the build produces a binary, but it fails like 100 out of 600 of tests.

A

This is due to the c plus brexit. Exceptions are not propagated through sequels on arm platform, at least by default, on the clan compilers.

A

We needed to pass the dash f exceptions to compile c code. This is this was very important because our http codex depends on this behavior uh we're we're on the path to remove the remove the exception from http 2 codecs, but that one was still at the issue. At that moment.

A

We also see some test flicks on arm64 platform. This is mostly due to different timing caused by different test timing.

A

Surprisingly, some tests runs faster on arm 64.. This causes the integration test faders.

A

So let's talk about the build performance. We use the aws r60g 8x large instance. This one has 32 cores and 256 gigabytes memory. We do cache with bazel remote, which helps a lot on the build performance without cache. This one cost like 40 minutes for every full ci job with the cache it normally runs within 15 minutes this, including like pull, pulling build docker image and producing test results.

A

For future development, we have some items left behind. One is web assembly support which is not merged into the upstream master yet, but it ex it currently excludes arm. 64. webassembly is a really important feature, so we we will need to add the webassembly support to arm as well.

A

Also uh there are some downstream builds, doesn't have arm 64 support, yet, like is still proxy or get envoy. We will work on this soon.

A

Thank you for listening to this talk. If you have any question, I'm in the on the platform to answer the question- and you can also ask me on twitter or slack.