Cloud Native Computing Foundation Cloud Native Rust Day EU 2021, 14 May 2021

Previous Meeting

Next Meeting

⏯

youtube image

►

From YouTube: Allocating Less: Really Thin Rust Cloud Apps - Evan Chan, UrbanLogiq

Description

Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon North America 2021 in Los Angeles, CA from October 12-15. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Allocating Less: Really Thin Rust Cloud Apps - Evan Chan, UrbanLogiq

In this talk I discuss in depth one of the key selling points of Rust for cloud native apps: creating really thin and fast apps or infrastructure layers. To do this, it is important to review and examine the memory allocation choices that we can make in Rust, and how to optimize them and data structures to ensure our Rust apps can reach the ceiling they are capable of.

- Why use Rust for thin cloud apps?
- Review memory usage in Rust
- How to profile Rust apps for memory usage
- Where to look for memory allocations in Rust apps
- Improving rust heap usage through better APIs, improving serialization, getting rid of clones, flattening data structures, etc.
- The effect of switching memory allocators (ie jemalloc) and why you might want to, or not
- A study of JSON processors: memory usage
- Different benchmarking stats to use for memory
- Traits and trait objects: how to avoid allocations and speed up performance (enum-dispatch and friends)

A

Hi everyone, my name is evan chan, I'm a senior data engineer at urban logic and welcome to my presentation.

A

So urban logic, we are an online platform that provides uh insights and we use thrust and machine learning, amongst other things, to give really great insights for transportation, economic development and other community use cases.

A

So why thin cloud apps? If we look at the progression of technology through the recent years, we can see that we have moved from in the old days.

A

We used virtual machines to uh now we're using containers everywhere, and we can see technologies on the horizon that are coming on fast, such as serverless and web assembly, and what we notice in this trend is that cloud infrastructure is getting smaller, thinner and more concurrent right, so things that maybe didn't used to be quite as important such as memory and allocations are becoming more important, because when each unit is smaller, you need to be more efficient, and you know we also live in a data-rich world.

A

So we need to process more and more data, so that has to be more and more efficient and finally using less memory, it's more eco-friendly right.

A

So the the question for everyone here we're here to learn about rust is why use rust for thin cloud apps, and they are actually mainly reasons for which I personally came to ross. I came to russ from writing a distributed in-memory database called filodb, which was on the jvm, and um I came for the no compromise aspect of rust that it was so to me that you could get performance, safety and abstractions at the same time. Usually you know you have to choose.

A

You know one or two of these, and um I found that it was uh pretty much mostly true that you, you could have all three things and also with rust. You get great control over memory usage, your allocations, and you have many ways to you- could even opt out of using a standard library, so there's many ways that you could use less memory and that small profile makes it very appropriate for writing everything from apps to oss, kernel level, code, hypervisors and so on.

A

Right, however, to take advantage of it, you have to learn how to control and measure the allocations. So, let's dive right in how rust apps use memory. So, let's start by reviewing rust memory model really quickly.

A

On the left side, you see uh what does rust put on the stack, so they would be primitives uh structs. So you have a couple different fields, but but everything in the structure is fixed size.

A

You also have fixed size, arrays and pointers references two things on the heap, so you have some stuff to work with, and then everything else that is dynamic is put on here, such as vects, which are lists and arrays things like strings and other more complex objects, as well as some miscellaneous things that we'll go over later now. One thing is that russ does not have a garbage collector, so how does it manage memory now this? This is a really important point for folks, those of you coming from dynamic languages.

A

This is the biggest differentiator above rust.

A

What rust's promises is that, even though it does not have a garbage collector, it will track how your data is used through a concept known as a lifetime, so to keep track of when your data is created when it is borrowed and used, and it tries to prevent unsafe use and sharing of your data and it tracks. When your data is no longer used, so it knows when to free it right.

A

This talk is not going to be about this, but but this will be important to remember when we talk about allocating memory.

A

So, let's with that, let's dive into some basic data structures that are used a lot and how much memory they actually represent. So we start off with strings and vects. So remember effect is a list of a fixed type of item so on the stack they use on 64-bit architectures like the x86 and modern r machines they use 24 bytes the pointer would be eight bytes, and then there is one field for capacity, which is how much how many items this data show can hold like.

A

That is how many characters for a string or how many items for a vac and then there's a length which is how many items it actually has right now. So these are both global data structures. Now, then, the pointer would point to an area in the heap that actually holds the the items or the characters.

A

Now the complement to vects and strings are slice, pointers and string slices. Both of these are immutable. They are the same as the previous ones, except there are 16 bytes because you have one pointer and you have a length because these things cannot grow. So they just point at a location and they tell you how many items they represent.

A

Now, let's look at a more complex data structure, the hashmap so for a hashmap, this is a bit more complex.

A

A hashmap can be implemented using buckets right, so all of your items are hashed into a fixed number of buckets. Each bucket in turn, can have one or more items in it when there is a collision.

A

So what we notice here is that the list of buckets is basically a vector of a bucket and in each slot a bucket. You know that points to a bucket and then each bucket entry would be your key and value pairs for the items.

A

And if my hash map is a string key and a string index, that means that I'm storing as the key and value entry in each bucket slot, I'm using 24 bytes for the key for the key string, struct and 24 bytes for the value string structs, so that added together, you know, gives you 48 right and now, assuming that the bucket.

A

Let's assume that each bucket has like one entry for the case where there are no collisions, then, if you add in you know the pointer for the bucket, which is basically the bucket itself, is like effect right. It's like a list so that pointer actually takes up another 24 bytes, meaning the overhead is up to 72 bytes per entry.

A

um So one thing you need to be very, very aware of is that for more complex data structures, you have these nesting of pointers that can create a non-trivial amount of metadata. Now, if your items like are large, then maybe it is not a problem. But if your items are small, then you might want to be careful and think about that.

A

Now, where could you be allocating memory in your apps, so we'll go over each of these items in detail later, but um starting in no random order? um One item that could take up a lot of memory is sterilization. That creates a lot of temporary objects and others one is when you use traits or trait objects.

A

When you um we'll go over that in a minute and um anytime, you use clone and data structures and and so forth.

A

Now um you could look at your your app and go through and find out look at these users, but maybe a better way is to benchmark right. So there's two ways you could benchmark your apps right, so one is to do dynamic memory analysis.

A

This looks at you know, starting from t0 the time when you start your app what is being allocated how much and it would track this over time and it would figure out hey how you know what what is the even if you allocate memory and free it, you know where, where are we allocating freeing memory to most? Where is what's the memory churn? So that's a dynamic uh benchmarking.

A

It there's also static memory. Analysis a static heap analysis this you might be more familiar with. If you come from a gc language like you know, java and so forth, where, basically, you could have an analyzer that walks your heap and figures out hey for a given point in time. uh What memory is being used up? What is using up the most memory now there are tools for doing dynamic analysis.

A

One is called keep track. Another one is called d-hat and I'll go over examples of d-hat static. Keyboard analysis is a bit more difficult in rust.

A

We can get overall memory usage pretty easily using something like general control and we can actually diff memory usage. You can also profile data structure using something called deep size, but but there isn't really anything um comprehensive, like you have with the jvm, but I'll show you some stuff you can use for that.

A

So um remember in rust that usually the more things that you type like box, the more you alkane. So that's just a fun hint to remember so, let's now, let's go over uh some potentially exponential uses of memory and how we can help reduce it.

A

So the first thing is: look at your method signatures. What are we passing in? For example, do you see function signatures like this, where I pass an effect of string, so this is quite common. You know in order to process some string lists right. So at first appearance you might like. Okay, that's you know a nice signature, but there's two problems when you ask the caller to pass in a back of strings you're, basically forcing them to allocate twice right once for the back and once for each string.

A

Instead, if we're able to uh change the signature to uh point at a string string slices, which is the second signature there, where we have this, you know ampersand and unstring.

A

This gives the caller two chances to avoid allocations. One is that they can point at existing strings instead of allocating a new string. That saves a whole bunch of memory, and the second one is that they can pass in a string slice instead of a slice of strings instead of a back right. And if you want even more flexibility, you can change the signature to pass in an iterator which gives you the chance to for them to pass in even non-less data structures, there's anything that can provide an iterator for even more flexibility.

A

So that gives you flexibility and gives you a way to avoid allocations.

A

So the um the next thing is that we can try to flatten our data structures like vac of string backpack and I'm not going to go over all of these, but there's a bunch of crates that will help you there, such as a nest it.

A

That will save you a lot of storage if you're, if you're, trying to have a bunch of a list of strings, for example, and there's a whole bunch of crates that can help you with strings that are basically inlineable where, when you have strings below a certain size, they will be on the heat on the stack instead of the heat, as well as things like small small back.

A

So there are ones we're doing smaller data structures, and I did a test using a repo that you can feel free to visit, where I show that by using nested instead of effect, you can save. You know like say 25. um You know total memory allocated.

A

So another area is by reducing clones. You might notice. Many of you are writing code using async. This is a really popular feature of rust. Now you can write code that um you know forks off work and you can do it away to wait, wait for it, which is which is great. You might find yourself, however, having to clone a lot of data structures when you're calling your async functions and these enclosures, because the data that is passed into a sync, because it's a future and could run on another thread.

A

It needs to be thread safe. One there's some quick tips, one is to consider using arc instead of clone, and this is something that makes sense, especially for things like lists things that, where you could pass in a lot of items, clone will usually do a deep clone.

A

What has to do deep, clone is to clone every item, so that could be quite expensive. Using arc does cost you a an atomic. You know couple atomic operations and it saves you a lot of memory. Another idea is to use something like an actor pattern. This is where you try to keep your state local instead of passing your state around, and so you keep your data structures within each actor or equivalently within each thread, and you use channels to communicate and you pass.

A

You know small messages and events. Only so that's a pattern that can help and it has other benefits as well. Finally, we consider using something like cow, for example, if you want to escape strings such as you want to um like for urls or something else where a lot of times, the string is not changed, but sometimes you need to create a new copy. Well, instead of creating a new copy, every time you can just copy only on right right.

A

Well, so how slow is arc really right, like in case you're, worried about using arc instead of cloning? Well? Well, if the data is anything concise, it's you know, it is fast, it is, you know, almost always faster. uh Actually, but basically arc is just an atomic increment on the clone and at the top of the document on drop and roughly on an x86. They estimate that this is between 30 and 120 nanoseconds, depending on which level of cache it might be faster and other hardware.

A

So now here's another area where you might be using memory is, you might find that you have a signature like this, where you're processing some item- and you know you want to pass the different implementations of traits right. So you make your signature, have this dying keyword for dying my trade right and now in order to be able to pass that function? Usually you need to box it, which means you need to allocate some heap memory for that.

A

Unfortunately, that means that every time you're calling this method you're you're doing this allocation, which is you know, not the fastest thing right so, especially in a hot loop one trade, uh one crate, sorry that you can use that helps a lot is called enum dispatch, which is really great.

A

What it does is, if all of your trade implementations are within your control, you can make it in any num, and all of your implementations of my behavior in this case are in this. Enum might be really numb. What we do is that we would annotate it with enum dispatch and in them this patch will will magically it will tie in with the trait and will actually make it so that your enum will implement uh the trait methods if all of the variants of the numbers also implement.

A

So basically you can change the signature here. The process might be really numb and you can still call my method on it on all the bearings that pass in this. This is a tremendous performance boost and it reduces allocations too. So this is really really great. I love it and I use it in one crate of mine, so another area where we could allocate a lot is are with sterilization right. So let's look at a quick sample, sturdy json with the deserialized from raw json to an intermediate value type. This certi json value thing.

A

This is quite common for serialization libraries and then it has to do another step. It has to take this intermediate ir and create a you know, say struct or something right. So um one way that you could go with this uh is to uh use. There are some faster crates, such as json rust, where the intermediate representations are more efficient, like json rust, has a short value type where short strings are on the stack, so this makes it faster and use less memory.

A

You can also go to binary protocols, although many of them have the same problem, they need to translate to some integrated layer or something, but but some of them can translate directly to you know, say a struct or something like that.

A

However, I think the best strategy is just to avoid civilization altogether, no serialization. So what does this mean evan? This is what you ask: what does it mean? What we mean is using something like flat buffers you might have heard of it: captain proto apache arrow.

A

It does take some work to actually create these formats, but what is usually meant is that there's no deserialization meaning once I create a flat buffer, I can send it over the wire.

A

When I get it, you know I can actually examine the flat buffer directly from the network buffers and extract values out without creating another into you know, without translating and deserializing it to my final form. So this is really fast it you know. Usually you can do no copy or no deserialization. This is really really good and I highly recommend it.

A

So, just an example from prostate json that sorry that um in this case, uh using my laptop and again, the comparison is available in this repo that I have and I used d-hat for heat profiling.

A

What we find is that using json rust, where it would reduce the maximum heat used, so d-hat actually measures how much heap is used at the point where, in the max of when the heap is um the largest in your application runtime, and we can see that it is quite a bit faster too it's like maybe 30-some percent. You know one-third faster because it has to allocate less uh and again.

A

This is the technique of using the civilization where uh it uh uses a stack value, a short string right and if you used no zero legition, it would be much faster than that. um However, uh for some reason the total allocations you know does not uh go down. This is like basically allocate so on and just to show you. I think this is uh good to show you what the d-hat output looks like.

A

um It basically gives you the top notes at a certain time, but you can look at what are the top allocators for all time and it will tell you it will give you a stack trace here. We can see that the stock trace you can easily trace it to 30 json when you know basically creating json objects. That is using up. That is 90 of your allocations, and it will tell you things like what is the average size for the allocations and the lifetimes that on average, is 73 microseconds.

A

So it gives you a lot of really useful memory: profiling, information, okay and just really quickly. We'll talk about a few extra memory allocation topics in russ. You can switch to memory allocator.

A

There are two popular alternatives to the standard. Allocator one is j malloc which came from uh well. It originally came from bsd sorry, but it was popularized by facebook and was created for reducing fragmentation and concurrency.

A

It does have a bit of overhead in terms of memory used, but it is faster than the standard allocator. Another one to check out is called we malloc from microsoft. It is designed to be a small, secure replacement for malloc and in practice I do believe it is also faster, so you can check that out.

A

And I have a benchmark that shows that sometimes it is faster.

A

Okay, finally, for certain use cases you can use bomb arena allocators.

A

And um usually this is when for special cases where you know, let's, let's say uh you want a sandbox, some memory for a part of your app. You know for queries in a database or certain namespaces. That kind of thing you can just take this elegant memory by bumping, pointer and then you can free it all at once. So for these things there is a crate called bumpalo, which is which is really great, and that can help sometimes when you want to control memory use.

A

Finally, you might be like evan, so reducing keep allocation is great, but I want to actually make my binary smaller.

A

You should check out the cargo bloat crate. This will analyze your rust, binaries and figure out. Where is your space being used, but there are tons of ways that you could actually reduce. You could get down to really really small like well below megabyte binaries. You can check out this and by the way the slides will be shared and you can actually click. You should be able to click on them.

A

There there's a url, so there is a blog that gives many things for reducing the size of binary, including stripping reducing debug things optimizing for size instead of speed, and uh if you really want to remove the standard library, that's a way you can get down to extremely small, like c size things, but be warned that that has a lot of trade-offs and I'm not quite sure if they are worth it. But it depends on use case.

A

So, um thank you very much. You can feel free to reach out to me on instagram, sorry, twitter, instagram, github, etc.