OpenZFS 2018 OpenZFS Developer Summit, 21 Sep 2018

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: ZoL Releases by Tony Hutter

Description

From the OpenZFS Developer Summit 2018
Slides: https://drive.google.com/file/d/0B_J4mRfoVJQRMzNPb1hBZ3RYcGg4NzFBSHd1clk4enVob01r/view?usp=sharing

A

Our next presenter is Tony hunter from Lawrence Livermore, National Laboratory, oh yeah, so from Lawrence, Livermore, National, Laboratory and Tony is going to talk about the release process for the various CFS versions. So right now, I'm gonna talk about released for Linux, 1.0 right and hopefully we gonna have a timeline on that anyway, all joking aside, the Tony is gonna talk about what goes into that thought process of what goes in the release and what kind of things that are being considered. So please welcome Tony.

B

Thank you, so yeah I think some of you've seen this before this was a presentation. I gave it the CFS user conference back in April, but it all holds true today. So yeah I want to go over CFS releases. I! Don't want to talk about the CFS release process, how you can get your code into the next release.

B

The CFS release, timeline and I also want to go over just some general stats about our ZFS on Linux user base, so shameless plug I work at the Lawrence Livermore National Lab we've got over 100 petabytes of disk running ZFS. We run ZFS underneath lustre so on. Our older machines were running lustre to five with CFS, oh six, five eleven I think and then on. Our newer systems were running lustre to eight on top of ZFS, oh seven, nine. So we usually run the latest release branch and we're starting to transition to lustre to ten.

B

So when I started putting together, this presentation, I talked to Brian and he found a really cool script that could go through all the stats on the ZFS on Linux page or go through all the server logs on the ZFS on Linux page and put together a bunch of really cool stats and graphs. That I think you guys are gonna be interested in and the first thing we did was we wanted to see how many people are using ZFS on Linux.

B

So what we did was we looked at how many people were downloading the repo MDX ml and we looked at how many people were doing that everyday and we multiply that by seven, because we know that Y M updates once a week and what we found was that there's, roughly 28-thousand updating, ZFS installs out there, and this is only for the OSS. We support so only CentOS and fedora, and this is not count Ubuntu gen to Debian.

B

Nor does it count anybody! That's updating locally, like at the lab. We update from a local server but I think that's a pretty healthy number.

B

If you look at which repo MDX ml they're downloading, we can figure out what OSS people are using and what we found was overwhelmingly.

B

People are using CentOS, specifically one of those CentOS 7 flavors, but at about over 75%, and this is broken out to different ways in this crap, by unique visitors and by unique hits and I, don't know which one is more accurate, so I just included them both, but they both kind of tell the same story and then the next thing we did was we geo-located all of the IP addresses that are downloading from the website and we were able to break it down by region and I. Think I thought this was really neat.

B

You can see the biggest users are North America, with 45%, followed by Europe at 31%, unknown at 12%, so maybe IPS that can't be geo located, followed by Asia, Oceania, South, America and Africa, and then I further broke those into the top three regions, and so here's a graph of North America and again the y-axis is the percentage of worldwide unique visitors. So you can see that 40% of all visitors to the page are from the United States, followed by Canada, and then I basically took the top ten here.

B

But the interesting thing was, even though these other countries at the end are tiny, they're still there and what you find when you actually look at these stats is that most countries, even no matter how small they are, how what the population is. There's at least one download of ZFS.

B

So it's fun to look at those you're. Looking at Europe Germany's, the top at seven percent, followed by UK at five Russia at about two and a half France Netherlands Sweden at about one and a half, then Spain Italy, Switzerland Poland. So it's a bit more distributed in Europe, looking at Asia Middle East Japan's, the the biggest at about three and a half percent, followed by China a little over one percent, South Korea, Taiwan, Israel and everyone else's about a half percent or below and again nearly all countries are represented here.

B

Like you'd, see Iran, and here you see two rock, so the other thing I looked at was okay. Most people do a yum install ZFS, but how many people are actually building from tar balls and what we found was that between two to four thousand people download the tar balls per release- and the other thing I noticed- was that very few people are downloading the sha-256 signature. So I guess people trust us.

B

This is a graph showing the oh seven, six release and I just wanted to show here that you do see a spike right when the release comes out right about there. So a lot more people are watching and downloading the latest releases. The other thing you see here is that these troughs are the weekends, so that tells me that most people that are using ZFS on Linux are using it professionally and they're downloading it during the rather than guys just working on their nas boxes at home.

B

So this is a graph of our commit frequency, so we average about 11 commits a week about 1.6 a day and I. Think that's a pretty healthy amount, we're not just flooding github here, and these are the top 10 committers of 2017 I. Think at least half of them are in this room.

B

One of the interesting things was that in 2017 there were only 48, unique emitters and I thought that was really interesting, because you've got 20,000 people using ZFS on Linux, and then it's just a very small group of developers that are making that happen. So it just shows how useful our work is to everybody else.

B

Here, I made a graph of the a7 release branch, both the RC and the regular release branch and originally I made this graph with all the ZFS releases, but it was so long that it just wasn't even visible on the slide, so I broke it down into oh seven and the takeaways here are the oh seven or oh seven.

B

Oh, our C branch took about ten and a half months to stabilize so keep that in mind with Oh 8 RC and then I typically do the O's 7 stable releases every six weeks, or so, with the exception of the last release.

B

So you can kind of get an idea of when to expect the next release from here.

B

So when do we put out point releases? Well, there's no fixed schedule, it's more on demand and the main reasons we do. It are there's a new kernel version that breaks all the interfaces and we have to fix them again. That's number one, a new version of Fedora or CentOS, which usually comes with a new compiler. That's a lot more strict and complains about a lot more things or our patch set for the next release. Just gets too big and we just say you know what we just need to put out a new release.

B

There's too many things, people want in there or you know, there's too many bugs that we need to fix.

B

So how do you get your code in the next point release, so you put out a pull request? It gets accepted. It goes in the master branch. So what you do is you go to your pull request and on the side here on the right side? There's a projects link, you click on that and that will bring up this project's dialog box and in here, you'll see the latest release, and this is an old slide.

B

So it's oh seven, eight, but the latest branch would be in there and you just click on that, and what that'll do is put. This pull request into a queue for us to review, to pull into the next release.

B

And so this is the project page. This is a list of all of the pull requests that we're expecting to put in the next branch. So, if you're wondering okay, what's going to go in, oh seven, eleven, you could go look in here these we don't limit only to these pull requests. It could be that one of these pull requests has a couple dependencies that we need to pull in first.

B

So we'll pull those in you're welcome to put any pull requests you want in here, but the things that we're gonna not accept are things like very large, potentially destabilizing packages, we're looking more for pork, wet or bug, fixes or really small patches that are benign or really small features that are pretty benign.

B

So this is a very old slide. I, don't know if all of you saw, but the Oh 8, o rc1 branch was released on Friday.

B

So there's a lot more in it than what I have here. These are some of the heavy hitters device. Removal metadata allocation classes- dear aide, unfortunately didn't make it, but the one we're really happy about merging the SPL and the ZFS is in yes,.

B

And we've already merged, like encryption, sequential Rhys, overs channel programs, there's a lot of big features that are already in oh seven. Oh, that will be a no seventh or Oh a doe as first-class citizens. So I think people are happy about that and I'll be happy because it'll be really easy to cherry-pick all the new patches into the oh eight one release.

B

Okay, these are the links to the releases page and the project's page, just for completeness I did want to talk about the process that I go through to put together a release because there's a lot of stuff I have to do so.

B

Basically, when we want to put out a release, I'll open up that projects page and I'll go through all the patches and I'll look for all of the commits hashes I'll put those in a text, file sort them by commit date and then I'll start cherry-picking them in and a lot of them are going to conflict because we don't have all the code from oh seven in the release branches.

B

So I try to fix all those conflicts and typically I, do all this on the latest fedora like 428 and then once I have all those pulled in I'll. Do a sanity build on Ubuntu with the absolute bleeding edge kernel to make sure that it'll build against the latest release kernel and then also build it on Fedora CentOS 6 to just make sure that it'll build on an older OS, then you'll see a pull request from me.

B

Come up with all the patches cherry-picked in basically saying like okay guys, this is what we expect for the new release, and people can comment on that and say you know what I want you to pull this patch in or you might want to wait for this patch to go in first and take that and the other putting it in a pull request is that you get to have buildbot test everything after we're happy with that pull request.

B

Let me check my time after we're happy with all that then I'll package, everything up on a build server that we have so we have an internal build server that you copy the code over there run a bunch of scripts it'll build all the packages for fedora, currently 26 through 28, and sent us six and seven it'll build all of the RPM repository data, I'll sign all the packages, I'll sign the metadata and then I our sync that over to a staging server- and this is basically the equivalent of what you guys do- a yum update from so I test.

B

Everything out on there I load up a bunch of VMs in Amazon, so all the ones we support, fedora and CentOS I'll, make sure I can do a yum, install I'll, make sure I can upgrade from an older version and then I also have a test that will try to download the source code and build everything from source.

B

So that's what goes into a release and Bryan has been working on a way to build the packages and build bot. That's going to save us a lot of time. This is something that we're working on that we want to do for the next release so that we don't have to do as much as many of these manual steps, and it would be also be nice. If, for any given pull request, you could have the RF PMS available.

B

So like a lot of the other projects out, there they'll build nightly rpms, and we don't do that. But that would be really nice to have and I think that would make it a lot easier for people to test. So that's it again. On Friday we released Oh 710 and Oh 8 Oh RC 1. So please bang on those and install away any questions.

C

B

B

Seven, oh yeah, it was right after the oh seven, oh I can't remember what happened there. Maybe something broke I.

C

Can't remember its.

B

B

I gotta repeat these questions, so the question was we're going to do Knightley's only for our PMS and I mean we're still working on it, but right now it's just our PMS, because that's all we officially support.

C

The other question I had was you talked about here, it's not being quite in. Is there like.

C

B

Don't know the status on D raid right now you could ping Brian D I, don't know if you have an update.

B

And for those online they were saying that number one D rate needs to be rebased with the latest code and a lot of testing.

D

So how's it working with upstream or I guess downstream distributors and CFS homiletics like.

B

As far as I know, with Ubuntu, they take one of our point releases and make it work. They make their own packages and kind of do their own thing, and so they're, not really updating. With all of our point releases.

B

Any other questions.

B

The question was: can you build binary packages for fedora instead of DKMS? You can.

B

Because it's easier just to do the DKMS yeah I mean I, know what the lab we, we don't use, fedora we sent to us and we use the K ABI packages. So that's like incentive for doing it, but you you can do you can build a KA bi of fedora releases.

B

B

Yeah I don't have a good answer for that I mean.

B

Yeah, so the question was: why don't we bump the meta file on master, but we do for the release, branches and I guess the answer is like we probably should do that.

B

Yeah, so for those online that the oh seven one you're saying came out because of a problem in the denotes multi-threaded allocation of D nodes, you said.

B

Interaction between large Tina's and multi-threaded alligator, okay, good memory.

B

Yeah so you're saying that the oh seven, oh and the oh seven seven were the only two that could cost corruption and that's why there were pointed leases right after them. Yeah that's correct.

B

All right anyone else, yeah.

B

Sure so, like I said, once I've cherry picked, all the patches I put out a poll request for that and that will run through Bill BOTS. So basically it gets the whole standard, buildbot testing and then, after we put together all the packages, I have a script that goes and tests to make sure that you cannot install out the RPMs on all the different platforms that we support and build from source.

B

The question was: is there any plans for performance regression, testing infrastructure and I'm, not the person to answer that you could talk to Brian afterwards.

B

Answers there's plans, but not concrete ones. There's always plans all right. Anything else, all right. Thank you very much.