Rust Programming Language RustFest Barcelona 2019, 29 Nov 2019

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: RustFest Barcelona - Pietro Albini: Shipping a stable compiler every six weeks

Description

A new version of the Rust compiler is released every six weeks, with lots of new features and bug fixes in it. So far the Rust team managed to ship stable releases on schedule with almost no regressions, ensuring everyone can upgrade their toolchain without fear of broken builds. This talk explains how our release process works and the tools we developed to catch regressions before our users do.

https://barcelona.rustfest.eu/sessions/shipping-a-compiler

A

Pietro: hi, everyone.

B

In this talk, I am going to shed a bit of light on why we on how it works and why we do.

B

It that way., as they said, I'm pietro,, a member of the rust release team and a colleague of the infrastructure.

B

Team., I'm actually a member of other teams,, but I do a lot of stuff in the project.

B

I think everyone is aware by now that we actually got our release out a few days ago, with some.

B

Features everyone waited for, for a long time, but that's not the only release.

B

Six weeks later, we released 1.38, which was released in september and changed a lot of.

B

B

users just reported 5 regressions after the release.

B

Two of them just broke valid code.

B

Six weeks earlier, there was another release, and this changed tens of thousands of lines.

B

Of code- and we just got three regressions reported and unfortunately all of them broke.

B

Valid code,, but it's a very little number.

B

Even before, we got 1.46 out in july with just four regressions supported, so I wanted.

B

To explain a little bit why we do releases this fast, which creates a lot of problems.

B

For us, and how can we prevent regressions, and just get very few reported after the release.

B

Is out?, so why do we have this schedule??

B

The question is interesting because it's really unusual in the compiler world.

B

I collected some stats on some of the most popular languages, like why there are some.

B

Efforts to shorten the release cycle.

B

python recently announced that they are going to switch to a year. Schedule.

B

Rust is sort of popular and has a six-week release. Cycle.

B

In a compiler world, that's pretty fast,, but there is a simple reason why we do that.

B

We have no pressure to ship things..

B

If a feature is not ready,, we have issues,. We can just delay it by a few weeks and nobody.

B

Is going to care if it's going to get stabilised today or in a month and a half.

B

And we actually do that a lot.. The most obvious example is a few weeks ago, when we decided that async/await wasn't ready.

B

Enough to be shipped into rust, 1.38 because it turns out it wasn't actually stabilised.

B

When the beta freeze happened and there were blocker issues, so we would have to crash the.

B

Future and that's something that we would not love to do.

B

We actually tried long release cycles and it turns out we don't. Actually it doesn't.

B

Work for us. the addition came out. The stable edition came out in early december and in september.

B

We still had questions: how to make the module system work?.

B

We had a proposal at the end of september, which was not implemented yet and that's what.

B

Actually was released,, but the proposal had no time, users didn't have much time to break.

B

It., it broke a lot of our internal processes..

B

We actually did this thing, which is something I'm not comfortable with still,, which is we.

B

Actually learned that a change in the behaviour of the model system directly on the beta branch,.

B

Two weeks before the stable release came out, and if we did a mistake there we would have.

B

No way to roll it back before waiting for the next edition, and we don't even know if.

B

We are going to do the 2021 edition yet.

B

This broke almost all the policies we have, but we had to do it because otherwise we would.

B

Not be able to ship a working condition, and thankfully it ended great.

B

I'm not aware of any huge mistakes we made, but if we actually made them it would be really.

B

Bad because we would have to wait a lot to fix them and we would be stuck in the 2018.

B

Edition with a broken features set for compatibility, reasons.

B

So, with such fast early cycle,, how can you actually prevent regressions from reaching.

B

The stable channel?

B

of course,, the first answer is the compiler's test suite, because rustc has a lot of tests.

B

We have thousands of them that test both the compiled outward, but also their messages and.

B

We have builders for each poll request which takes three to four hours, so we actually do.

B

A lot of testing, but that's not enough, because a test suite can't really express everything.

B

That rust language can do..

B

So, of course, we use the compiler to build the compiler itself.

B

Because for every release we use the previous one to build the compiler.

B

Like on nat, we used beta, on beta, we used stable and on stable. We used the previous.

B

Stable. that allows us to cut some corners because the compiler bases use some unstable features.

B

And also it's a bit old, so there are a lot of quirks in it, but still that can't catch.

B

Everything. we get back reports from you, all.

B

We get them mostly from nat,, not from beta,, because people don't actually use beta.

B

Asking our users to test beta is something we can't really do.

B

With such a fast cycle, we don't have time to test everything manually with the new compiler.

B

Every six weeks; and languages with long release cycles can afford to test the beta release,.

B

But we can't, and even when we ask, people, don't really do that,, so we actually had an.

B

Idea., why don't we test our users' code ourselves?? This is an idea that seems really bad and seems to waste a lot of money, but it actually.

B

Works and it's crater.

B

Crater is a project that brian anderson started and I'm now, the maintainer of which creates.

B

Experiments which actually get all the source code available on crates.io and all the rust.

B

Actually tested for every release to catch.

B

Regressions. for each project, we run cargo test, two times, once with stable and one with beta and if.

B

The cargo tests runs on beta on stable, but fails on beta. Then that's a regression and.

B

We get a nice colourful report where we can inspect them.

B

This is the actual report for 1.38 and we got just 46 crates that failed and those are.

B

Regressions, nobody reported before.

B

Then manually, it goes through each. I hope we didn't break any of yours and manages.

B

The log and then files the issue, so then the compiler team looks at the issues, fixes them.

B

And ships the fix to you, all.

B

1.39 went pretty well, this is 1.38 and we actually had 600 crates that were broken so.

B

If we didn't have crater, there is a good chance. Your project wouldn't compile anymore and.

B

This would break the trust you have.

B

We know it's not perfect., we don't test any kind of code because we don't have access to your source code and.

B

Nobody got round to write, scrapers, yet, and also not every crate can be built in a sandbox.

B

B

It's not really something we can scale forever in the long term because it uses a lot of.

B

Computer sources already,, which thankfully are sponsored,, but if the users sky rocket.

B

We are going to reach a point where it's not economically feasible to run crater in a timely.

B

Fashion anymore.

B

Those are real problems, but for now it works great.

B

It allows the types of regressions that often affect some thousands of crates and it's a.

B

Real reason why we can afford to make such fast releases.

B

Without it,, this is my personal opinion,, but I know it's shared by other members of the.

B

Release team,, I wouldn't be comfortable making releases every six weeks without crater, because.

B

They would be so buggy, I wouldn't use them myself.

B

So to recap, the fast release cycles that we have allow the team not to burn out and.

B

To simply ignore deadlines, and that's great, especially for a community of mostly volunteers.

B

And crater is the real reason why we can afford to do that.

B

It's a tool that wastes a lot of money but actually gets us great results.

B

So unfortunately, I don't think I have time for questions,, but I'm going to be around.

B

The conference today, so if you have any questions,, you want to implement support for the repositories,.

B

Reach out to me,, I'm happy to talk to you, all, and thanks.

B

Actually,, you have some time for questions.

C

Pietro: oh,, that's great.,.

D

Do we have time for one question or two,, because I have actually two.

D

Pietro:, I don't know that.

D

okay,, so first question, you were hinting that maybe the addition idea wasn't such a.

D

Success for us. would you think that jeopardises a possible 2021 edition of the language?.

B

Pietro:, like the man issue, wasn't really the addition itself;. It was that we actually started.

B

Working on it really late., so basically we went way over time with implementing the features.

B

This is my personal opinion,: it's not the official opinion of the project,, but if we.

B

Make another edition, I want explicit phrases that we won't accept any more changes after.

B

This date and to actually enforce that because we nearly burnt out most of the team with.

B

The edition. there were people that were just for months, fixing, regressions and fixing bugs and that's.

B

Not sustainable,, especially because most of the contributors to the compiler are volunteers.

D

Okay., I will ask also my second question., so the second question is: for private repository,. Of course you cannot run crater,, but how could.

D

Somebody who has a private repository,, a private crate setup,, would run crater,, or is that.

D

B

Pietro: someone could just test on crater and create bug reports if they could face.

B

B

we have some ideas on how to create a crater for enterprises, but it's just a plan,, an idea,.

B

And at the moment we don't have enough development resources to actually do the implementation.

B

Test and documentation work that such a project would require.

D

So that would be necessary to support crater in a private environment?.

D

C

Okay. any other questions?.

A

A lot of crates have peculiar requirements about their environments.

A

Can you talk about how crater handles that, and specifically, is it possible to customise.

A

The environment in which my crates are built on crater?.

A

Pietro:, so the environment doesn't have any kind of network access for obvious security.

B

Reasons,, so you can't really, you can't install the dependency yourself, but the build environment.

B

Runs inside docker. we have these big docker images, 4gb,, which just has our usual packages installed,. So.

B

If you know your crate doesn't work, you can easily check with docks ls, because since recently.

B

It uses the same build code as crater,, so if it builds on docks ls, it builds on crater.

B

As well., and if it doesn't build,, you can file an issue,, probably on docks. Ls is the best place, and.

B

If I want 18 of 4 packages available,, we are just going to install the package on the build.

B

Environment and then your package will work.

E

Hello, thanks for the great talk., how long does it take to run crater on all of the.

E

Pietro: okay,, so that actually varies a lot because we are making constant changes to.

B

The build environment, changes with machines and such.

B

I think at the moment, running cargo test on the entire ecosystem takes a week and the.

B

Running cargo checker,, which we actually do for sample requests,. If there is a poll request.

B

That we know is risky,, we usually run it beforehand and then do cargo checker, which is faster,.

B

Takes about three days,, but that really varies mostly because we make a lot of changes to.

B

A

C

C

Is it possible to supply the crater run with more runners to speed up the process?.

C

Pietro:, I think we could.

B

At the moment, we are just in a sweet spot, because we have enough experiments that we.

B

Fill out the service,, but we don't have any idle time.

B

And the queue is not that long.. If we had more servers, then the end result is that for a bunch of time the server is.

B

Going to be idle, so we are just wasting resources and we have actually a sponsorship firm, co-operation,.

B

So if we reach a point where the queue is not sustainable anymore, we are going to actually.

B

Get funding,, though crater is really heavy on resources.

B

At the moment, I think we have 24 cores and 38gb of ram, 4 telebytes of this space,, so.

B

It's not something where you can throw out some old machine and get meaningful results.

B

Out of it. all right,, we are out of time for questions.

A

We are going on to the next talk. thank you.