Rust Programming Language RustConf 2021, 15 Sep 2021

Previous Meeting

Next Meeting

⏯

youtube image

►

From YouTube: RustConf 2021 - Project Update: Libs Team by Mara Bos

Description

Project Update: Libs Team by Mara Bos

A story about how things get stuck, and how to get things moving again. Taking improvements to the standard library's locks as an example, we'll take a look at how large efforts often get stuck, and explore what can be done in such situations. We'll look into breaking problems apart, how to narrow the scope of discussions, and finally at some organisational changes that might help a bit.

A

Hi, my name is maga and I'm going to tell you a story, a story about things, getting stuck a story about the standard library about large changes and small changes about 300 common github threats are running out of energy on how to move things forward when they're stuck a story about focusing on one tiny thing at once: chapter one problems.

A

Once upon a time there was a standard library and that standard library needed some synchronization primitives, most important of all the mutex, its name stands for mutable exclusion and a mutex is used to provide mutual exclusive access to something usually some data or some other resource such that it can be safely shared between threats.

A

The idea is simple: you first have to lock it, and only then do you get mutable access to the thing it protects when you're done, you unlock it allowing others to lock it again.

A

If you lock it while something else already holds the lock, your thread goes to sleep and the lock method only returns once it actually exclusively locked it.

A

So a mutex is a way to share something between threads, allowing them to mutate it, while still upholding the fundamental guarantee. There is only ever one mutable mutable reference to something at once.

A

There are more synchronization primitives than the mutex, but those are not very important right now. What is important, though, is that implementing these primitives yourself is hard bugs in lock, implementations are often very subtle and very hard to find and debug.

A

They are a complex interaction between very low level and sometimes even architecture, specific code and the operating system.

A

The operating system has to be involved because threats need to need to be put to sleep and woken up and scheduling is a responsibility of the operating system.

A

But luckily operating systems already shipped with an implementation of these on linux, mac and other unixes, that's part of pthread, which is part of supposed standards on windows. They are called critical sections or on newer versions of windows. We have the improved srw locks or slim read, write logs.

A

So, given that they are already implemented for us, can we just wrap them and be done with it?.

A

Unfortunately, the answer is no: these were designed for c and they don't exactly match the requirements of rust. There's many very subtle things. You would do differently when designing an implementation for rust, but I'll highlight just three of the bigger problems.

A

First problem in rust: if you own an object, you can move it to a different place in memory, or at least when it's not borrowed.

A

That means an object cannot assume it will stay at the same memory address over its entire lifetime. These lock implementations, however, do not allow their objects to be moved here. Is the documentation of srw locks, for example, and it clearly states an srw lock, cannot be moved or copied with most actual implementations today, moving would actually work perfectly fine, but their specification or documentation simply doesn't allow it. Future implementations might work differently, so we have to follow these facts.

A

So as I work around, we allocate them on the heap. We put them in a box that way if you move the rest, mutex object. The actual underlying system detects does not move this works, but it is inefficient because now there's allocations and interactions involved and it means we cannot construct these objects at compile time. So you cannot just have a static. Mutex object because needs to be allocated at runtime.

A

Second, problem rust, wrong usage must be safe. In rest, we have this very well-guarded border between safe and unsafe. There are plenty of things that are wrong or unwanted, but still perfectly safe in terms of memory safety.

A

A program that always panics or loops forever is probably just plain wrong, but it is perfectly safe.

A

Let's take a look at an example: we have a mutex protecting some integer and we lock it and then, sometime later before, unlocking it, we try locking it again on the same thread. This will result in a deadlock. It will wait forever until the first lock is released, which will never happen because we're stuck waiting.

A

This program is definitely wrong, but it is safe that locking is safe.

A

Unfortunately, the os locks do not always guarantee a deadlock in this situation.

A

If you relock a beta, admit x on the same thread, the behavior is undefined, or at least when using the default settings, which was a problem in older versions of the standard library and critical sections on windows are actually re-entrant. That means they purposely do not that look when you relock them on the same track. It will actually succeed.

A

So if we go back to the example, we can take a look at what would happen if we, if we would just wrap those mood taxes in rust, we lock a mutex and we try to lock it again on the same track. But this time we do not get a deadlock, it might succeed.

A

Then, afterwards we have locked the mutex twice, both lock methods promised exclusive access, and now we have two mutable references to the same data. This breaks a fundamental guarantee of the rest language resulting in undefined behavior.

A

And finally, a third problem in rust, leaking or forgetting things is safe by definition again, usually not something you want, but perfectly safe an example. We have a mutex, we lock it and then we forget that we locked it. It never gets unlocked again. The mutex stays locked forever later. The meter gets dropped, while it is still locked.

A

This is safe because the compiler knows that we have no access to it anymore. The only thing that had is forgotten.

A

These locks, however, may not be destroyed while locked here's, a screenshot from the petra documentation and it clearly states attempting to destroy a lock mutex results in undefined behavior.

A

Conclusion rust is not c, and retrofitting rust's concept of ownership and safety on top of existing things is hard in very subtle ways. Rust is fundamentally different.

A

Trying to fit the c-shaped peg into a rust-shaped hole, doesn't always work making it actually fit requires force and frustrating workarounds.

A

Chapter 2, a solution on a friday morning in september 2018 three years ago, alex from the library team then brings up these problems on the internals phone. He also suggests a potential solution. Replacing these locks by parking lot parking lot is an idea that originally comes from webkit.

A

The idea is that if you have a way to put threats to sleep and wake them up, or as they call it park and unpark them, you don't need anything else from the operating system to implement your very own locks.

A

The parking lot is a global data structure, where threats go to sleep and wait which keeps and it keeps track of what they're actually waiting for it is implemented using low level operating system primitives, which vary a lot per platform.

A

Once you have such a parking lot, you can implement your own minimal taxes without having to think of the operating system at all.

A

There is a rust crate called parking lot, a very popular one made by amaneo, which provides a rest implementation of this idea, and not only does it implement such a parking lot but, most importantly, it implements mute taxes and other primitives on top of it.

A

Its smooth taxes are just a single byte and require no allocation or other runtime initialization.

A

That means they can be constructed at compile time, so you can have them a static variables too, that locking or forgetting, is well defined and there's no way to cause and define behavior with them. If you stick to the safe interface, so this seems like the solution to basically everything we talked about.

A

There was a bit of discussion, but people are positive and not much later on a cold and rainy saturday morning, at the start of december, linus fernstant opens a pull request: a pull request, embedding the entire parking lot crate is the standard library and changing the standard library to use it for synchronization primitives quickly. Afterwards, several people began to review the code discussing potential issues.

A

More and more comments were posted, which were all quickly addressed, but this continued into the new year in january, and discussions went well into february and then march and more in april and still in may and even in june and july discussions were still going on and if you scroll a bit further, you see november popping popping up.

A

So what's going on. What's what's blocking this.

A

One of the problems was simply that it is a huge amount of new code, or at least new to the standard library.

A

Not only is it a lot, but it's also the kind of code that requires very specific knowledge of academics and all kinds of operating system details.

A

There is very few people who know all these details of the operating system, let alone of all operating systems.

A

An example on linux parking lot uses a syscall named foodx. It's a very low level function that is also used by pthread implementations to implement their mood taxes.

A

The standard library had never used the syscall before, and some questions came up about, whether it's okay to start using it, which versions of linux support it. Is it maintainable? Is it easy enough to understand this wasn't really a big blocker, but it wasn't yet another topic. On top of the already large amount of things to discuss on older versions of windows where necessary, where the necessary functionality wasn't available, it used something called anti-keyed events which are undocumented.

A

It's it is not as bad as it sounds, because they're still well known and only used on versions of windows that do not change anymore, but it was just yet another discussion to have, and then there were stability guarantees.

A

Switching to parking lot would mostly be just an implementation detail, but there were some very subtle things that would be publicly visible and might be considered a breaking change. These were probably not a big deal, but yet another thing that needed to be discussed and needed a decision.

A

I could go on with this list. I'm not even sure if I know the full list, but the overall issue is clear. There are just too many things to be discussed at once.

A

The author and other people had tirelessly worked on addressing all the comments and issues and a lot of progress was made, but after all this time, activity on the threat was rapidly slowing down by the time yet another year it started. There were just a few more comments until eventually everyone just ran out of energy.

A

The momentum was gone, the pr was closed without getting merged, leaving behind a massive discussion. Log of almost 300 comments.

A

So what is our conclusion here?

A

It's sad and frustrating when a large change ends up like this, but it's very normal and it happens all the time things are always harder than they seem, and the only way to move forward is to just try it and see how it goes.

A

So I guess what I want to say is large changes push things forward, even when they fail.

A

Even though this full request was not merged, it had a big impact. Things in parking lot had been improved because of it, and we now know a lot more about what needs to happen to improve the standard library's logs.

A

So a next attempt should be much easier now.

A

This brings us to chapter three stuck. We are now in the year 2020 and at this point we are in an interesting situation.

A

What's happening now is a weird kind of deadlock in the progress towards better mood taxes, a kind of deadlock that happened surprisingly often in the rust project- and I bet in many other projects as well what's happening now. Is that every time that someone brings up some small problem or some improvement to the current logs people reply with oh, but parking lot is going to solve this.

A

It makes changes to the existing logs very low priority, because people are still expecting parking lot. Integration to happen sometime soon, people are hopeful, but at the same time, if someone then asks what's happening with parking lots, when is it going to be integrated and what needs to happen?

A

Everyone just gets silent and points at the massive discussion threat from before. There's no good overview of what the problems were. There's no list of changes that need to be done. There's no clear place where to start, and just like that, a failed attempt becomes an obstacle.

A

While a previous attempt with meaningful discussion should have made next attempts easier, it also, somewhat surprisingly, can make it much harder to start to start a new attempt.

A

Someone working on a new solution would first have to overcome this obstacle. They have to read through years of discussion, to know if their attempt is making mistakes that have been made before and in which ways their attempt is different.

A

This takes a huge amount of time and energy, often more than someone is willing to spend on it.

A

So we now have on one hand, people are assuming that this parking lot integration is still going to happen soon and that results in small things getting blocked or shut down, and on the other hand, we have new attempts at parking lot, integration that are just not happening, and over time we have less and less context of what is actually what it actually got stuck on.

A

If you ask five different people of what has changed got of what this change got stuck on, you'll probably get five different answers, because no one really remembers all the contacts anymore.

A

The conclusion here is not that anything anyone did anything wrong here. We just need to find better ways to recognize situations like this and then find a way to handle them.

A

As for recognizing this, I I hope this talk will help a bit for handling them. That's a lot harder, but I have some ideas. Let's start small chapter 4 tiny steps, instead of trying to gather all the contexts and reconstruct the entire web of all the different issues that were involved, you can take a look at one single obstacle at a time pick the tiniest obstacle you can find and try to move it out of the way, try to find a way to fix it in a way that does not touch any other parts.

A

I'm going to quickly talk you through a few examples of small obstacles. I try to get out of the way a few small changes which I hope will make a future attempt at a large change in this area, more likely to succeed.

A

Obstacle, 1 stability guarantees the standard library. Documentation, makes a lot of promises which we usually never try to break. um That's part of our stability guarantee.

A

There's one small promise about mute taxes, however, that we would not be able to keep if we switch to a more efficient implementation, whether that's parking lot or something else, I'm not going into much detail now, but we basically promise to panic in certain situations with newer, mutex implementations. We can no longer always give that panic in those situations.

A

It's not a big deal because, instead of panicking, it will just work and it's very unlikely that someone actually relied on this panic happening. In fact when it was brought up during the original big parking lot integration. Pr, the reviewers seemed fine with this change, but it will be something that will be brought up next time. Digging up the old discussion then, would help, but it will still be a distraction.

A

So I sent a very simple pull request that just removed that specific detail of the promise I described the context and the entire team signed off on it, agreeing that this change is fine, after which it was merged.

A

So obstacle 1 gone one fewer distraction. Next time it would have come up.

A

Obstacle 2 movable mutaxes, because the system mutaxes we use are not movable, we put them in a box on the heap new implementations such as parking lot no longer require that which is great, but we didn't have a mechanism mechanism for keeping the box on some platforms and removing it on others.

A

That would become relevant if we don't want to or cannot switch to a new mutex implementation on all platforms at once.

A

So I worked on making it possible to make this decision for each platform individually to make it possible for potential future improvements to be made per platform, allowing for smaller changes to have a reason to do so. I looked for a first platform to unbox the mutexes on to no longer put them on the heap.

A

Unix platforms were out of the question, posix doesn't allow it and there are many different implementations that all need to work with rust on windows. It's also not allowed to move them. The documentation is very clear about that.

A

But then I found out microsoft's documentation is actually on github and, more importantly, it accepts pull requests. So I just send a pull request. Adding the promise that we need basically just adding a note, a clarification that if the lock isn't borrowed you can move it.

A

And, to my surprise, just a few days later, it was actually reviewed and merged, and now it's part of the official documentation, I'm not entirely sure what I expected, but I did not think it was going to be this easy anyway. Now that we now, we actually certainly have a platform, a very big one, with movable mutexes.

A

So then I went back to the to the rest, repository and added the mechanism to allow unboxing mutexes on windows while leaving other platforms untouched.

A

I have not been able to benchmark it, as I don't have a windows computer to test it on, but I'm assuming it made a noticeable difference to the speed of the rust mutaxes on windows, at least in the very contented cases anyway. Now that it's merged. That means that next time we want to unbox the mutex on another platform, it's just a flip of a switch so obstacle 2 removed.

A

Obstacle 3 new operating system, primitives.

A

Efficient mutex implementations such as parking lot use a very low level of functions of the operating system, specifically on linux. One would use the food access call and on windows one would use the wait on address function.

A

Neither of these had been used in standard library before a point of discussion that came up in the parking lot. Pull request is whether we are okay with using them, because we have no experience with them yet so I figured if we can just use these apis in another place in the standard library. In a more isolated, simple case, we can get this obstacle out of the way.

A

Since they're so useful for threat parking, I figured we can just implement the standard library spark and unpark functions with it. These functions are not used much and they are more primitive than what a parking lot provides, but they are there and pretty simple to implement with these os primitives without touching any other code.

A

So I send two pull requests: one to implement them using food taxes on linux and another one to implement them using weight and address on windows.

A

The first one was quickly merged. The second one took quite a bit longer and had quite some discussion discussion that would have otherwise happened on whatever other requests would have introduced it at this api.

A

It was stuck for almost two months, two months that a future attempt at improving mood taxes would have gotten blocked on otherwise, but eventually this one got merged as well. So yet another obstacle gone conclusion.

A

Smaller chunks are easier to chew.

A

I am not saying that everything can be easily split up in smaller chunks. Not at all. You can't just take a big, complicated thing and cut it into some arbitrary small pieces and hope it works.

A

But if you do see a way to break things up, it can really help focusing discussions and can really make it easier to make decisions by just focusing on one thing at a time it does not solve any complicated problems, but it does make it easier to focus because then the smaller issues are no longer distracting from the bigger issues.

A

Chapter 5, moving forward after contributing for a while, I was invited to become a reviewer and then later team member of the library team.

A

At that point I started to think more about what can be done less in terms of code and more in terms of process and organization.

A

Other teams have successfully been doing something called mcps major change proposals, a lightweight way for someone to get feedback on something big. They want to do before doing it.

A

Big changes to the implementation of parts of the standard library could really use this too.

A

However, the library team was mainly focused on the public interface and its stability.

A

The implementation of the library had been the responsibility of the compiler team for a while, although they also didn't have much bandwidth for it, they would handle critical issues, but wouldn't have time for guiding larger changes.

A

At that point, we decided to rename the library theme to the library api team to highlight that their main responsibility is the public api.

A

And that's when I created a new library team to focus on everything else, library related.

A

So now we actually do have a team to handle library, mcps and more and now the api team can guard stability and work on api design without getting distracted by implementation. Details we're still figuring out the best way to do things and what our responsibilities should be. But so far things are going pretty. Okay,.

A

Far more important than the teams, though, are the contributors. Teams just exist to facilitate to guide, to make decisions when necessary, but most of the work is done by the contributors.

A

Our main task is making sure that those contributors can well contribute with as few obstacles as possible.

A

There are many things that are far from perfect. Contributing can be a frustrating experience. We often have to say no to things, but we should be careful to not scare contributors away.

A

We should guide them towards things that we can actually say yes to. We should recognize if things can be split into smaller parts and we should try to clean up after failed attempts make sure that context does not get lost over time. We should recognize when something is blocked on something and figure out how to unblock it.

A

This is all easier said than done, especially considering. Most of us are doing this in our free time and we all have limited bandwidth and energy, but as long as we just keep making steps small steps forward in the right direction, then we can achieve anything eventually.

A

Thank you for watching my talk and enjoy the rest of resconf.

A