Jenkins Platform SIG, 18 Jun 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Platform SIG 2020 06 18

Description

Jenkins Platform Special Interest Group meeting JUne 18, 2020. Topics included Google Summer of Code progress on the git plugin performance improvement project

A

Sounds great great welcome everyone. This is the Jenkins platform special interest group. It's June, 18 2020 thanks for joining us. Let's look at the at the agenda and be sure that we agree on topics and then we'll work through the agenda. So what we've got is we'll talk about open action items.

A

Then we are delighted to have rishabh with us to talk about his progress on the get plugin performance, improvement, Google Summer of Code project, interesting results, good results and fascinating progress. So we'll give him as much time as in each ask questions, record, notes, etc. Then, Alex and I are gonna talk briefly about the adopt open, JDK for docker image change and worked through what it means. Do some discussions briefly and we'll we'll conclude: there are there any other topics we should add to the agenda.

A

No, okay, great all right, so let's do the review of action items. I! Have the action to switch the meeting URL to use the CDF zoom account I've done that Rishabh. You will notice that you can share the screen without me having to do anything. That is one of the results in switching to use. That's great! Let's zoom account I'm not hit. I had still had the open action item to open a Jenkins enhancement proposal for docker operating system. Support, I apologize, yes, I will get that done.

A

We are in progress right now on the docker, build rework PR or a derivative of it. So so that's encouraging!

A

Oh no I! Take it back. The one I'm in progress on is the next one, so we still need to make some further progress on the docker. Build rework PR, because this one gives us a better structure Alex. This is the docker manifest and parallel build changes that I haven't done yet anything. Yet what to report.

B

There I'm starting to take a look at it running local, builds and things like that. So far, it's looking good the way. I apparently have my duck or authentication setup. It's it's I'm. Getting a 400 won at some point when it's trying to publish because.

A

Okay now- and this is one that since I, don't think I have authorization to do anything to the docker hub account and I hope I, don't I should be able to do the evaluation as well and run it in some sort of local mode. A dry run mode.

A

B

I think the dry run is still there great.

A

All right, so this this still needs review. There's lots to do in in docker image work and the next was the Alpine image update PR and that one is PR 956 here that one I would had started review on it yesterday last week, some additional review yesterday and further this morning so.

A

Alright anything Alex you wanted to report otherwise on action items, any other topics there I know: ok, Rishabh I think we're ready for you on the get plugin project. Ok,.

C

Thanks for, let's.

A

See I need to stop sharing, don't I. Here we go. Yes, all right. All yours.

C

Ok, a gate plug-in for improvement, as Marcus mentioned, I'm the G software 24 student, the shop is my name so the objective of the project. So we it's a simple objective: to improve the performance of gate, plugin and they've identified two ways to do it. The first is to you use jmh, which is a framework java micro, benchmarking harness which provides us a safe environment to test benchmarks.

C

So we want to use benchmarks to identify known or unknown performance issues with the existing gate implementation we have, that is the sea like it and the Java pure Java implementation G again. So this is the first objective we have created. We have already created gate, fetch meant parts and we have some great insights from those benchmarks and we are going to implement those inside the gate plugin. The second is to fixed the existing performance issues we have in the great plugin and I'm going to discuss them further in the presentation.

C

So what have we done and what are we doing? The first was the first thing we have done is to integrate jmh the modules inside the gate, client plug-in. So basically, what I've done is that I've added the benchmarks to the test module of the plug-in and now the benchmark scan done on CI tank installed IO, which gives us a wider selection of platforms where I, where we can run the benchmarks and have a comprehensive result, result profile and how it looks. It's it's basically like this, a pipeline, a sample pipeline.

C

So we we build the repository. Then we check it. We land the benchmarks and then we we we have a JSON which is a report which we can use to feed in into visualizer, where we can see the benchmarks visually, how things are going in how the operations are performing. So this is the first thing we've done.

C

After this we have tried to fix the redundant fetch issue. So what is the issue and how was it affecting the gate, plug-in performance or is it affecting actually not was so? The issue is very simple. It's the reason behind it is not simple, but it's it's. So this is a freestyle project.

C

I am trying to check out a repository, so, as you can see here, I am trying to fetch the upstream changes from the repository once, which is expected, which is normal, but then again, I am doing the same thing incrementally so so this second fetch is redundant and you've actually written test cases to understand it.

C

If we go through some scenarios to understand if we're losing or breaking in use cases, if we are avoiding this 2nd fetch- and we we've come to the conclusion that, though we have some interact, interactive testing left right now, but still we have come to the conclusion that we can remove the second fetch, but we will not show that if, if the removal of this second fetch is going to improve the performance of the gate plug-in B, we did not have concrete numbers to show that to support that hypothesis.

C

If this is an app hypothesis that removing the second fetch would actually improve the gate, plugins performance, so for that initially I created a benchmark for benchmarks, actually with git fetch. The benchmarks were simple: the first benchmark is a simple, a single git fetch with an arrow respect, which basically means that it is trying to fetch one single deposit, one single branch, which is the master branch. The second test was also a baseline, single git fetch test, but with a wider web spent.

C

That means all the branches, then the third and fourth tests are double fetches with the same thing: narrow respect and a wider ref spec. So what before even fixing the issue? What we could see from the benchmarks was, as you can see, this is an interactive chart as I progress with test1 test2, test3 and tests for for each repository. The execution time increases when, when I'm moving towards test three and test for yes mark okay,.

A

So so, what we're seeing here is as you're iterating through the the horizontal dot that moves across that's telling me repository size. Is that what it is the the sort of test? One is smaller. No, no tell me again whether test on to okay for okay.

C

Test1 test2, test3 and test for are basically benchmarks, test1 and test2 our benchmarks, which calculate the execution time for a single, get fetch as three and test for our benchmarks, which calculate the execution time for double get fetches. What you seen here, the the x-axis is the repository size and the y-axis is the average execution time which is increasing repository size. I have four repositories: the sizes report for repo one it's less than one empty for the second one. It's 5 MB for the third, it's ninety and for the fourth, its 300.

C

What the while we're iterating the dots, it's basically running through the benchmarks, I have done and the results of those benchmarks. So what you can see here is once I go from test one two to test three: four: with repository three and four, you can see a remarkable increase in the execution time and with repo one in two. There is not much of an increase which kind of tells us gives us a hint that, with a small repository size, the incremental fetch would not add too much performance overhead.

C

But with a larger repository size, there is a possible chance that the second fetch would be adding considerable performance overhead, so this was kind of a theoretical experiment before fixing the fetch which which gave us a reason to actually remove the redundant fetch now.

C

So how did we fix it? It was a simple fix.

C

We added a boolean to check once we're using this first, which we will not use the second search, and so once we did that, are we seeing any change in performance and to see that I used profiling filing with Java flight recorder in on JDK 11, so I ran the Jenkins war with the updated gate plugin with my changes, with my fix for the redundant fetch issue of the Jenkins war on JDK 11 with JFR Java flight recorder, which is which is a profiling tool, comes with the JDK 11. It has very low performance overhead.

C

So so what you can see here is so I took two repositories: one is the Jenkins I/o or 40 MB size, and when I took the Samba get a positive, it is near about 300 MB. So, with these two repositories, I do I perform the simple check out build, and here what you can see is without the fix for the smaller repository. There is not much of a difference. It's it's! Basically a 10 second difference if without fix, it's 2 minutes 55 seconds and with the fix it 2 minutes 46 seconds.

C

So there's a 10 second decrease in the execution time, for the gate, fetch calls for the plug-in overall and with the second bit of much larger repository, 300 MB size. There is approximately a two minute difference once we remove the second fetch, which is I, think a considerable degree improvement in the performance and if we increase the size of the repository and I'm sure we will have much larger differences. So so I've done this once or I'm done done. This actually twice to confirm, if is my before my performing results, profiling results, correct or not.

C

Maybe I need to do this with a wider variety of repository size to confirm that ok, redundant, fetch is actually the removal is actually improving. He performs, but I think this is yeah. Yes,.

A

Ma'am, that's the gorgeous, so so this was with command line, get as the implementation or with jagged as the implementation come.

C

On learned, you excellent.

A

Okay, so then your Java flight recorder could could not even could not have any impact actually on the on the actual operations performed by CLI get because it's a separate sub process. So your that gives me even more confidence in your benchmark numbers and your measurements, because alright there's there's not even the profiling from the Java flight recorder component, basically pauses or or has just sub process or just process impact, not it can't touch the get command-line process.

A

That's a C program running entirely separate, very nice, excellent, okay, so the so at 300 that large large repository, but that's really cool.

C

So so for this issue, this is this is what we've done. So the next step forward for us is to implement the performance improvement inside the plugins, and to do that, we've figured out two steps right now. The first is to provide a compatibility switch to the users. The switch is going to be basically for users. We were assuming that once we once we add the improvements. Whatever we get from the benchmarks, we had those improvements inside the plug-in. There might be cases where the users functionality might be affected.

C

Performance might be affected in some ways we do not be. We did not anticipate, so we are providing a switch, and it's going to look like this roughly I. Have it it's in the configure system in Jenkins, taking inside the gate, plugin, you will have a checkbox which says, revert, performance, improvement changes so once it will revert to the old version of okay. So so this is the first step.

C

The second step is to actually selectively switch between the implementation, that is, the C or jig it and from the benchmarks we found out that forget Fitch. We found out that the size of repository is is the is the biggest Vidya is the biggest parameter which is affecting its performance for an example. One insight we have is that for for a depository less than 5 MB size, J kid is going to perform better than get and for a larger repository, get is considerably performing way better.

C

It's performing way better than J get so so, if we want to implement that inside gate plug-in, we need to estimate the repository size before creating the client, and so we have we've had some discussions on using heuristics like using gate, LS remote to calculate the number of branches we have without cloning, the repository inside the plugin and- and we can use a rough estimate so that we know maybe 50 launches means a large repository or a lesser number means the smaller size could possibly use. That second could be.

C

Another option could be to use existing rest. Api is exposed by github or gate lab where little while the size of the repository. So we could also do that. We were thinking how to do it. Currently, it's in process and any suggestions on how to do this. How to estimate the repository size without joining the repository first would be appreciated.

C

So this is what we have done and what we plan to do and any questions.

A

Okay, so so the first, the first cycle, then, is the redundant fetch. You've already got a target where I say we know that there's a significant performance improvement to be gained here and this this redundancy affects both command line, get and Jake it so it's redundant in both cases. It's not it's, not just one or the other they're both doing this redundant fetch. Yes,.

C

A

So everybody gets benefit if, if no matter which implementation they chose or no which implementation we were to choose, they get benefit potentially large, potentially small, but they get benefit by this. This this choice: good, okay and then oh, go.

C

Ahead, no no he's going to do then.

A

On the on the the benchmarking, the jmh, based benchmarking, the the challenge there was that you've got to figure out before knowing before having performed the full operation, you have to decide which implementation should should actually execute the operator the operation. So before doing a fetch, you need to make a decision, shall we use CLI get or shall we use use J get yes.

A

This thank you very much that I think think you've covered it well, Alex did you have any questions you wanted to ask or let's see, is Oleg joined us if.

B

They're so with the JSON file that's created, is there any way of visualizing that in Jenkins itself as a way, I get a snapshot? Yes,.

C

I, look there. There is an already-existing jmh, visualizer, plugin I haven't tested it. I haven't integrated it with the Jenkins instance, but I have to test it, but it what it does is it consumes D JSON file and I. Maybe I have just one. Second, how it looks I can show you.

C

It's called geometry, port and.

C

Okay, I cannot see how the visualization is made. I don't have a tear, but um there's a site. It does the same thing. The plugin is basically used this website to do so, and I.

C

Have a jmh JSON sample just bear with me for a second writing.

C

Topic, so this is how the visualizer visualizes the JSON.

C

So once we add that plug-in without process of building the benchmarks, then we can see the results like this.

B

Okay, often well.

A

B

An infra a plug-in installed on Sierra Jada dream player right.

A

Well, and- and another variant here, is that one of the benefits of see ila Jenkins that io4 Rishabh- is that, whereas he's got his Mac Mac OS machine that he runs base checks on CI de Jenkins, that IO provides him access to arm 64 to AMD 64, to s3 90 X and to PowerPC.

A

So so he can safety check that hey are the benchmarks that we're using representative even across the different environments, that CI a-- jenkins thought io represents oh and windows, yes, I forget there's one other platform there and we we regularly have surprises with Windows. So, yes,.

B

So then the second question I have is: is the integration of jmh stuff? Is that something that can be easily added to? Maybe the parent, palm and and optionally enabled for all plugins? Or is there a plan to do anything like that? So.

A

It's it's oh go ahead. Rishabh I should not answer. This is yours.

C

It's okay, so, yes, there is an option that the Java the Jenkins test, harness already as it has the jmh harness inside the dependency. So once you have so there's a version here for the poem you need to have for Jenkins. For that particular dependency, once you have your grid, I, don't remember the version I think it's 2.5 for the Jenkins test, harness once you have that you, you don't need anything else.

C

Just you can use the j much benchmarks inside you are plugging in any plugin that was done before it was done in a previous jisub project. The old strategy plugin, okay,.

B

And does it require special tests can, like you, have to mark tests as part of the image data collection, III.

C

Can I actually show you? How is a benchmark? A so-so? Yes, like J unit test? It's it's it's in a it's a little similar like that, so you you annotate it with a benchmark annotation and you need a benchmark or ll as well, which will identify the benchmarks and run it with some options. You'll need the options can be how you want the results average time or the throughput time.

C

Whatever you want, how much you will warm up the benchmarks before you run them the you the time unit, folks, the JVM folks, you want so all of those options they are included in the runner and that class. So how do you run that so there's a profile in Mabon maven profile, gem, each benchmark, so your benchmarks will be well done from that command and you will have a generator JSON report from that command and if you want to integrate the benchmark, you want to run them on CI Jenkins Road.

C

Are you there's a there's already a base step called run benchmarks. We just have to add that in your Jenkins file and if you have benchmarks to it, they will be run on the CI Jenkins infrastructure.

A

C

Yes, Alex well.

A

Thank you, so we shop would you would you show the Jenkins file I I'm I'm enamored, with with how elegant the work from the role strategy plug-in made it for this, but then we found some some interesting issues that made me beg rishabh to do some extensions. So in the end we get client plug-in he's going to show us what the Jenkins file looks like.

C

Yes, so this is the step which enables running the benchmarks and we provide the name of the JSON. We want as an output so before so. What was happening before was that for every pull request for every branch, Marko screening for testing the benchmarks they today they have a considerable time duration. They run for maybe an hour. So for our depends on the kind of benchmarks we have it takes time. So it's an unnecessary addition for people who do not want to test the performance. Maybe so so.

C

What we've done is a simple pattern: matching that we will run the benchmarks only for the master branch, all for any branch which is related to G saw so it's pretty simple.

A

See for me for me, Alex, the treat here was that somebody else had taken the time. The the last year, roll strategy had taken the time to create this plugin. This pipeline, shared library, called run benchmarks, and so rishabh was able to without without doing anything more than using run benchmarks. He gets it to execute now on the CIO Jenkins at I/o infrastructure and then from for my needs. He also gets, though, the ability to put conditionals in there, so we don't have to run it on any branches except specific targets.

A

C

A

Work rishabh thanks very much. Thank you. Thank you and good luck with your your ongoing efforts. It's good I'm excited to be able to announce the the release of the get plugin that includes this. This will be so cool. Thank you. Thank you. Thank.

C

You so much mom, okay, so yeah.

A

And I'm gonna switch on screen share and, let's take a look at the next topic, it was that we want to do share screen really there. Okay, I think the next topic is around the Alpine image. Oh I didn't take any notes, Rochelle bout, all inserts and notes here on your your results, so that we've got them in the text as well. The recording of the of the meeting will be post shortly after the meeting. Okay.

C

Should I add them in notes, while you're, oh, that.

A

Would be great if you're willing to do it. That would be wonderful, I think so adopt openjdk for docker on Alpine, Debian, Buster and CentOS so Alex. This was for me at an excuse to just have a conversation with you to be sure that I've understood what what we're expecting in PR 956 and what we should be reviewing sure.

A

So as far as I can tell it's giving us a nice integrated new structure where we say we start with the jdk target is the first part of the directory name, then the operating system and then the B. What do you call it? B JVM, optimization technology, whether a hotspot or open j9? Yes, that's great, and so so then what I was trying to decode was. How does that map to tags in the repository? So, for instance, I was looking for a Debian squeeze based version of this and didn't detect one.

A

If is that, have I misunderstood, there's buster, so there's Debian 10 Buster in the slim image, but there doesn't appear to be a Buster. Another is Buster, but somehow or other I didn't find that label. Okay, so I! Oh, this is JDK. 11 I was looking for JDK 8 with Buster. So what you've done here is if using Java 11 I get the newest version of Debian, not the old version of Debian. Well,.

B

So this is just taking exactly what we had before oh and migrating structure, so it it doesn't like I'd only image new image, I added I, believe it is the is a JDK 11 Alpine everything.

A

Else is already existing ICO, so so the the directory structure is a refactoring. In that sense, not an increment of adding a whole bunch of new distributions. Thanks I had misunderstood that yeah.

B

And so the directory structure was changed and then the face of it was changed such that it uses adopt of JT right.

A

And so that's that's this year, which in particular with Alpine, is a real win for us, because it gives us a newer version of the Alpine based operating system right instead of 3.9, it's now 3.11 correct, which whatever they're in latest, is yeah exactly and it's since the alpine project has stopped stopped any any work or I. Guess it's the wrong way to say open JDK is a project, is no longer publishing, updates to the Alpine image right. It's it's stuck on Java, 8 212, as.

B

Well, as they're, they are not publishing, correct images or other architectures.

A

Okay, all right, that's a big problem.

A

Okay, so the the the transition here to adopt open JDK gives us a more current operating system base, a more current JDK, something that the the project adopt open. Jdk is actually actively testing and- and we get a better base image for ourselves.

A

Excellent okay! Thank you all right. So one of the one of my pleas to the viewers of the video is this is a great excuse to do additional testing to help us as we evaluate this because it's it so. The intent here is not to alter what we're delivering in the sense of based operating system version, but we are altering which operating system we're using or which, which base image we're using. So instead of using open, JDK will use, adopt open, JDK rich got it. Okay, excellent.

A

Okay, that thank you for that help. That was, that was what I needed.

A

If there are there any other topics, we should review today.

A

All right, let's call it called an end to the meeting Rishabh thanks very much excellent summary, great results and looking forward to further Alex thanks for your time as well. A recording of the meeting will be posted within about an hour hour and a half okay thanks mark thanks Alec, but.