Cloud Native Computing Foundation EnvoyCon 2020 - Virtual, 12 Nov 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Community & Extensibility: Building Envoy's Flexible Compression Subsystem - Mikko Ylinen, Jose Nino

Description

Community and Extensibility: Building Envoy's Flexible Compression Subsystem - Mikko Ylinen, Jose Nino

HTTP compression is used by web proxies to compress data before sending it out over the wire. This saves network bandwidth and speeds up transfers. Until its v1.15 release, Envoy implemented limited unidirectional Gzip compression. Through a cross-company (also cross-continent!) collaboration, compression was generalized to its own extension subsystem to make it possible to add new compression schemes, custom implementations of existing formats, and have fully bidirectional (de)decompression. This talk will walk the audience through the evolution of compression in Envoy, highlighting the flexibility of the extension system, and dive into two use cases now possible with compression subsystem: 0-touch bidirectional (de)compression between mobile clients and edge; and new compression implementations now possible, e.g., a compressor utilizing HW accelerators to optimize server compute.

A

All right, hey good day good afternoon, everybody welcome to listen to our talk about onward compression subsystem evolution.

A

uh In our talk we give a give the law you give the story of uh the kind of long but very successful project to redesign the new onboard compression subsystem with with some great community collaboration and then discuss some of its features and and future plans, um but before getting started, let us introduce uh ourselves first.

A

My name is mikko and I'm a software engineer at intel's, open source cloud software engineering team working on device enabling for kubernetes and cloud native applications in in this project. I've mostly been coaching dimitri, who did all the heavy lifting, but uh I've also, if you have unavoid contributions myself, mostly on the open, ssl side of things,.

B

My name is jose nino and I am an engineer at lyft, where I have worked on networking relating related technologies for the past uh four years. uh I was part of the initial open source team uh with envoy and worked on server-side technologies uh for the first two and a half years that um I was at lyft and then for the last year and a half I've been working on a related project called envoy mobile, which we'll discuss a little bit today.

A

Next, I want to introduce you dmitry. He is not in the presentation with us today, but his contribution to the work that we are discussing today deserves a special recognition.

A

Moving on to the story, how how how the unvoiced compression rework got started and what we learned from from the process.

A

But let's get started with with some common terminology and there are lots of variations used, are used across different code bases. So we wanted to explain the ones used in in this presentation and in in the envoy onward code base um within the compression subsystem. Overall, we have uh both compressors and decompressors and they compress and decompress data. Of course, naturally,.

A

So um and then, let's uh next, uh let's take a look. What the http gzip filter in in on envoy does and uh on the request path on the left hand, side that the client sends a request. The filter on on android looks for matching and gzip accept encoding header.

A

And the filter deletes it and and forwards the requests to the upstream service.

A

And when the filter receives the response data, it compresses the data using the gzip algorithm and finally, then onward proxies the gzip. Gzip responds back back to the client. So this is. This is how it how it works, and basically it is uh limited to only uh you know, response compression.

A

uh This all works. Well, if you are interested in providing compression based on just like one existing one, one, one implementation of of cheesy, but then you might ask what, if you want to modify that implementation say uh using hardware, accelerated gzip, for instance, or what, if you want to use a different compression algorithm altogether, like broadly broadly compression algorithm or finally, what if you may also want to decompress the data at the proxy proxy layer?

A

These were all the questions that remained uh and which we kind of asked ourselves, and that's one of the reason why we got started with the work that we did here.

A

um Well, let's look at uh some of the existing advantages in in envoy, um and these were kind of the things that allowed us to to start thinking uh the redesign, a more comprehensive way and design the comprehensive compression subsystem for quarantine.

A

And first of all, enroll is open source and in in our team. Basically, our motto is uh upstream for us. We wanted to propose changes to the upstream and and get them accepted and with open source. This is easily they should be easily doable.

A

um Second thing is envoy has very clear, abstract interfaces, for instance for compression. We had these very basic uh basic interfaces for compressing and decompressing data, and that is just uh provided through the data buffers.

A

And then, third, and perhaps the most important point envoy, has a very mature extension system that allows people to implement their own own logic for for very well defined extension points. And we can. We thought that perhaps compression libraries could be the first class extension points for online.

A

uh So with that, we just uh took the time to prepare the initial pull request uh to generalize the existing gc filter into a compression uh algorithm agnostic filter. uh Originally, we were expect we were expecting very smooth process in in up streaming, maybe with just uh a few comments review comments that we needed to to address, and then we should be done and complete with this with this contribution, but uh uh it did not go like that.

A

We everything did not go as as we expected and at some point, as you can see a long time since we originally started the purple request to work simply simply cut the stalls and we did not see a path getting getting the contribution accepted and myself and dmitry were like pretty puzzled. So why is it so hard to get this accepted, especially because uh the contribution the implementation was meeting, all the needs that many others also had asked in in the past?

A

And at this point we were, we were even like, considering going with, with the plan plan b and uh in implementing what we wanted for the hardware accelerator you're, just using custom external filters that meets meets our needs.

A

But then, luckily the work was spotted by michael and jose from lyft somewhere.

A

um I think in february time frame this year and they had seen the work and they also had the need for envoy decompression in envoy mobile to to keep their public apis with with compressed responses and then do decompression in on the client client side.

A

So it's uh it's not always just open source that makes these contributions possible, but especially for big features like this contribution that we had it's really about the community and how the community collaborates together to kind of complete the last mile and getting things accepted upstream.

A

And uh what what we learned from from all all these uh during the way to do while, while getting this, is accepted and uh merged in in upstream.

A

So um one observation is that uh sometimes it's best if you just share the big picture, the kind of the bigger the design idea in in a shared document uh to make the conversation possible and then uh eventually create the kind of consensus uh about the changes and the way forward before before investing a lot of time in in in working on on pocs and then scratching the work and starting all over.

A

But this document it also helps to build the community behind the work, which is uh super important, as we just discussed previously uh and one other thing. We also learned that uh sometimes all these open source contributions they they need a lot of patience and that don't give up too easily. If, if there's some delay in getting your changes, reviewed and accepted.

A

Good, so um next I think we have jose going to talk about generalizing, that the envoy onward compression.

B

All right, thank you mikko, so now that we have that background context of what existed in the android compression space with the specific gzip filter. Second, what were some of the use cases that were not possible with that previous implementation and then what advantages existed? That could make a general compression compression system possible in envoy and how all that work was possible thanks to not only a cross-company collaboration between us at lyft and the folks at intel, but really across continent collaboration with people in the united states and in finland.

B

So, let's dive into uh what we actually did in general generalizing, this compression system.

B

So the first obvious thing when looking at the g-step filter is that, um while this is what it looks from the outside, if you look under the hood, what you have is a filter in two parts, the hdp filter part and then the compressor part.

B

So the if, if we zoom in here, the htp filter part deals with the http level concerns like looking for the accept encoding header like miko, was saying or replacing the data frames with the compressed data, and then the filter itself uses the gzip compressor compressor, which it itself is the one that is compressing the data buffers using the gzip algorithm so uh going back to the existing advantages.

B

What became evident when looking at the filter this way is that there was nothing specific in the filter that required the compressor to behave in a certain way to use gzip to compress, because he used this very simple interface that we had seen before. A compressor only has one function: it compresses, so there was a very clear delineation of where we could divide the filter and then plug in a compressor library, regardless of the compression algorithm that it used.

B

So now that we had this generic compressor working within the generic http compressor filter, we could raise the compressor itself to be a first class extension in envoy, and so this is done using the classes that we see here where we have a class that derives from the config type factory class and then every particular implementation of of a compressor library derives from that in turn.

B

So now our gzip compressor, or perhaps our broadly compressor in the future, derived from the name, compressor library, config class and then the compressor filter uses this compressor factory class, the generic compressor factory class to create compressors with which to actually compress the data.

B

In other words, we achieved the first of our goals. We are now able to have any number of different compressor screen schemes in envoy without each having to implement an http filter themselves. We can just plug in any of them into the generic compressor filter, so whether it is a gzip compressor or a bradley compressor or like our friends at intel. Hardware, accelerated gzip.

B

All right so on to our second goal: what about a full compression system, uh one that can both compress and decompress data? Well, that was easier now that we had a generic implementation of a compressor filter.

B

We used the same principle of abstraction using the generic decompressor class and created a generic decompressor filter that used a portable generic decompressor, thus creating a fully pluggable hdp, decompressor filter.

B

So parallel to the compressors as first class extension points in android, we created decompressors as first class extension points in android.

B

Thus we pave the way for having not only a gcp compressor, but also having any number of the compressors to parallel the com, the compressors that we were adding.

B

So now we have a complete generic end-to-end compression solution powered by android, where service b can send a response, and then the http compressor filter can compress that response which arrives to service a and in turn its http decompressor filter, can decompress the response before proxying it to the service itself.

B

So, as you can see, compression is completely done in the network layer without having to involve any product engineers, owners of service, a or service b.

B

All right, so this is what the android compression extension tree looks like the common directory implements. The generic compressor and decompressor factory base classes that all compressors and decompressors derive from, and then each particular compression scheme has its own tree, where they have a compressor and a decompressor and potentially a common base class that derive from you know some scheme specific functionality.

B

So in android. Currently we have the gzip compressor and decompressor, but, as we have been saying, the beauty of this first class extension is that any number of compression schemes can be added, whether added upstream to the the main envoy, repo or in your private, build of envoy all right. So that's a whirlwind tour of uh how we generalize decompression uh but most exciting. We want to present some of the case studies uh that were enabled thanks to the to this generalized compression scheme.

B

A

B

First one: oh sorry, we could go ahead.

A

Yeah so just time thanks jose, so I was about to say that our first uh case study talks about implementing hardware accelerated compression for for onward.

A

um So some a little bit about the rationale for why why doing hardware accelerated compression, typically hardware, acceleration, can bring several benefits to to your application.

A

For instance, um as you know, being like very time consuming uh process, it might make sense to overload all this compression processing to a dedicated, co-processor, sometimes even like a hardware that is completely optimized for this particular task and then the added benefit with that is that then you can free cpu cycles for for something more important that runs on on the main main cpu and then uh the co-processor, the accelerated core processor, doing all the all the all the work.

A

In our case, we have implemented a reference implementation using this uh on onward compression subsystem for and for an intel, quick assist technology card. It's a it's got a dedicated card optimized for crypto and compression compression processing, and this implementation adds a qat compression compressor.

A

We have it uh implemented in an external repository, as also mentioned. It is also with with the envoy extension mechanism possible to add your custom implementations easily living in in their own external repositories and the qat compressor we have uh implemented uses a framework. What what we have called qat, zip and then the q, a d zip, then directly uses the q, a t, hardware, abstraction layer and the driver and sending out all the compression requests to uh to the dedicated co-cooperative co-processor.

A

We have it right now. We have it only for compressors.

A

Basically, we are able to compress the data and what we, what we are working on right now, is that we're just adding looking to add almost there adding the decompressor piece for the hardware acceleration as well, based on our our experiments this this piece of compression uh in, in addition to the fact that we are also able to free cpu cycles for some kidneys.

B

All right, so, thank you for that case study. uh There are some published results and we will share the link with the presentation slides. So now, let's talk about the envoy mobile use case, I want to first give a quick overview of the goal of the android mobile project and what it is as a baseline context for this case study. uh This diagram shows an increasingly common topology in network distributed systems.

B

These days, I'm sure many of you at your companies uh have a setup similar to this, where envoy is deployed as a universal network, primitive where it deals with the majority, if not all the network traffic in a company's architecture.

B

However, my team at lyft recognized that we had left an important hop outside the ecosystem, because traditionally, we've treated mobile clients as independent from the backend infrastructure, and we have built unique solutions to what was assumed to be a unique space.

B

So we identified a technology gap because, in spite of all the work that we had done server side, we saw that three nines of reliability at the server side is meaningless. If the user of a mobile client is only able to complete the desired product flows, a fraction of the time, so what we want side are the same guarantees that we had server side.

B

So that is what we created android mobile. We propose that we don't need to treat the mobile clients any different, at least from the network perspective, as we do with the server set infrastructure. So this is what my this is why my team built this over the last year. We have built bindings into envoy that allow us to run the code base as a native library in a mobile client application, and so we have great documentation and a ton of material about the project.

B

So, if you're interested in the project in itself, uh please uh feel free to take a look. It's also all open source, but today we are here to talk about compression and in particular show you uh with how, with android mobile in the client and with envoy, at the edge of our infrastructure, and with this new generic compression subsystem, we were able to do some things that would have been too resource intensive in the past.

B

So compression at the edge is particularly important because payload sizes can lead to performance degradation under adverse network conditions which are usually experienced by mobile phones, making public api requests to your infrastructure.

B

So, as I mentioned before, lyft compresses all responses at the edge and we wanted to have at least the same behavior when we migrated of our old client stacks to envoy mobile.

B

So in the past, before android mobile, implementing compression at the edge required three different code bases, one for the edge proxy one for the android client and one for the ios client. So this might be okay. If we just wanted to put you know, one compression algorithm in place and just leave it be, but my team started asking questions very similar to our colleagues at intel. What if we wanted to squeeze better compression and does better performance with different compression schemes?

B

What about bi-directional compression, where we not only compress responses but also requests, and these questions quickly become intractable? If for every variation that we want to test, we need to implement the same code three times, server, android and ios, and that is really one of the main reasons we embarked on the anway mobile project. To begin with, because by using the same code base everywhere with envoy mobile in the client and envoy at the edge, we are able to have only one implementation for every new technology.

B

We want to experiment with affording us on parallel consistency and extensibility at a fraction of the engineering hours required in the past.

B

So in other words now we do have truly a true universal network, primitive that we can experiment with and why we became so interested in collaborating with our colleagues at intel in this compression subsystem all right. So, let's see how we brought compression back to the edge uh with only one implementation and without involving either our client or server engineers.

B

So before we go into compression, let's look at the request flow using android mobile, very similar to onboard on the server, and we mobile proxies requests from the client to the edge proxy upstream and then the edge proxies the requests that receive that request further proxy, that to the upstream service.

B

Similarly, on the response path, the edge receives a response sent it to the client using android mobile, which in turn surfaces that to the application via callbacks in the library all right. So the first step in enabling compression is installing the new compressor filter, with a particular compressor library on the edge nodes.

B

At this point, nothing changes in the client. The client sends a request which envoy mobile proxies to the edge, but now the edges compressor, filter, searches for the accept encoding header, which allows the filter to know if it should compress the response for this request. But the header is missing, so it simply forwards the request and then, when the response comes back, it does nothing with it and passes it back to android mobile.

B

But now we can install the decompressor filter in android mobile, and here I want to highlight how easy that was. We didn't have a help, filter before and then boom. We installed the filter so easy. Well, jokes aside! This is easy because android mobile is envoy, so we can install the same filters that we use in envoy on the server, including the decompressor filter, all right. So now that we have the decompressor, the client sends a request and the decompressor adds the accept encoding header.

B

So when the compressor at the edge searches for that header, it finds it. So then the compress decompress, the compressor strips the header and lets the envoy proxy the request, but thanks to the accept encoding header, the compressor is now primed to compress the response, because it knows that the client is going to be able to handle this compressed response.

B

So when the response comes back, the compressor compresses the response and in turn the decompressor filter decompresses it before sending it back to the application all without the client or the server engineer being involved at all nice. So we were really excited with this. First, zero touch implementation and we started thinking what else could we do at the edge with envoy mobile in the client and with envoy in the server?

B

So potentially we could have bi-directional compression. So, instead of this setup, where only the response gets compressed to be sent across the edge, we could have a more complicated setup, but it is a setup where both the client and the server have both a compressor and a decompressor filter allowing both requests and responses to be sent compressed across the wire.

B

But we could, you know, do more than that. Not only are we allowed to do bi-directional compression now, but we could also run experiments with different compression schemes and fine-tune our compression settings to achieve less degradation at the edge and ultimately higher success with our public apis.

B

So this is the android mobile case study, and this very, very flexible compression system has really enabled us to pursue a lot of avenues that would have been intractable due to the engineering hours required beforehand. So those two were the big use cases that drove compression uh forward with a cross-company cross-continent collaboration to thank for.

B

However, we have many things planned for the compression space in envoy. First, even though the new compressor filter has been around since the 1.15 release, the old gzip filter is still there. We plan to remove it from envoy by the next release. 1.17 second is bradley. Support which is currently being worked on by dimitri at intel.

B

Third, is having a bi-directional compressor. Bi-Directional decompressor has already been implemented by myself, but to have a fully bi-directional end-to-end compression system we're missing, request compression. This is very easy to add in the compressor filter in case. Anyone in the audience is interested.

B

Having this pluggable compression libraries enables us to apply compression to other parts of envoy. Your ideas are welcome, on which parts exactly can benefit from it.

B

Lastly, it is worth mentioning that the performance of gzip compression in envoy still has room for improvement. For example, gc compression engine x works about 25 percent faster than an envoy when configured. Similarly, these are just some of the items we have planned in the compression roadmap, but we can't wait for seeing what other interests the community has for this new subsystem. Well,.

A

uh That's uh that's all we had uh for today, thanks for for listening and watching us and looking forward to see you in the community.