Jenkins Google Summer of Code Office Hours, 31 May 2019

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: GSoC 2019 Demo: Jenkins Plugin Performance testing with JMH

Description

See more about the project here: https://jenkins.io/projects/gsoc/2019/role-strategy-performance/

Meeting notes: https://docs.google.com/document/d/1iiErNAIBM9JK1Lni-QsCr7BxytnWZxdOzq_DgbASdY0/edit#heading=h.c8jquq4z82ms

A

B

Now the recording has started so hi everyone. Today we have my first demo of G, so performance improvement project. We will one of the first objectives. Interest coding face is to have a benchmarking framework so that we can lead changes and every day will show the current state and possibly build. It has something to show so and I think it's great, so yeah a lot of meat here, I've made you a presenter, so you can stop at any moment.

B

C

Hi, everyone I'm a pivot, and so today, I'm just showing you about the benchmarking framework that I've created to run Java micro benchmark harness tests. So how does it work so it works directly from a J unit test. So it's easy to just run from your ID and we also added the functionality to run it automatically. I mean the classes which contain the benchmarks are detected automatically when they are annotated with gmh benchmark and for each folk that jmh creates a temporary Jenkins instance is started now to configure the instance.

C

We can either use the Java code as in the functions inside your plugin or the Jenkins scores itself, or we can use Jenkins configuration as code. Both of these are there in the role strategy plug-in now, and so how does a benchmark? Look like her? So basically, what you have to do is create a static in a class which extends th benchmark state, so this damage benchmark state is a state that is used by Java micro benchmark hardness. That is this is the only way you can pass information to to a running benchmark.

C

So what this class does is that it starts up a temporary Jenkins instance. It's almost exactly the same like what Jenkins test harness does. So what we can do here is like if we have some specific setup for your own benchmark. You can just over at the setup method here and if you want to do some cleanup, there's the teardown method and to get the Jenkins instance inside the benchmark at the bottom. We have the benchmark here.

C

So what we can do here is you can either use the Jenkins instant from the state, that's provided from the state or you can just directly use Jenkins that get instance, and after that you can put your benchmark whatever you like, so we can now when you use configuration as code, we just need to do the same thing except we just need to extend the configuration, ask our damaged benchmark state and to give the path to the Yambol file.

C

We just need to override this method and the Jenkins is configured automatically now the setup and teardown methods are still available. The only thing you'll have to take care now is to call the super method. First,.

C

So the benchmark reports they can either be evaluated from inside the day unit tests the damage benchmark when run returns, a Java object, so the values from the can either be used to fail or accept the test. Otherwise, the reports are written to the disk in Jason and these reports can be directly fed to the JMS report. Plugin I did share some examples in the mailing list. I haven't added them here yet, but I'll do that for the final presentation.

C

So jameth allows us to measure in multiple measure, multiple things, so we can measure the average time for a lot of operations. So how do much works is that it takes some. It creates multiple folks and it creates multiple folks and so for each folk. It first runs some warm-up iterations to warm up the JVM, so it doesn't have to compile the code again. I mean inline compiled to the machine code and after that it runs the iterations.

C

So the average time of these iterations can be measured and there's also throughput the number of operations and the single short time without any warmup. So there multiple things, so you can just see the giant samples, and so some of the challenges that I faced was these bugs, so test crumb issuer was which I was using because it was used in Jenkins rule from Jenkins test harness that was not allowed to be marshal from the configuration as code.

C

So what happens here and the other thing another again discussed on the mailing list- is that when we stop a jetty server, it leaves some threads behind. So what jmh does it is? Is it waits for after the benchmark has completed it, wait for these threads to terminate for 30 seconds, but then they do not they're in the waiting State. So Jamis has to forcefully kill forcefully kill the JVM, and these are the pull requests.

C

B

You for the quick introduction, so we, it would be great if you could briefly show how it works for all strategy playing I mean yeah.

D

B

A look at the tests which we already have as benchmarks and maybe even bringing them.

C

So, okay, what we have heard is a benchmark runner, as you can see, it's just part of the standard, a unit tests. So what we can just do here is just run it from here, so we currently have a lot of benchmarks which do take a lot of time. So, just let me read you this. Let me remove them from being discovered automatically so yeah. We just have this benchmark running, which uses configuration, as quote so.

C

This is all of the checks that we are making so just to make sure that it's being configured correctly and after that we are checking the time it takes to generate an ACL object. So just let me run it.

C

So it's just starting the Jenkins instance here this is the first folk.

C

This is the configuration, I scored, doing its work, and so now the benchmark would have started yeah, that's 70% CPUs with so it's using two threads currently so, as you can see, first, the warm-up iterations are run, and after that we should be able to see the actual iterations, which would be measured.

C

Yeah, so it will do the same thing again and it will terminate this fork start a new fork of the JVM and do the same thing again, and so after that we get a report position report.

C

This is a sample, Jason and so I'll just show you the this is what exactly the JH plug-in dessert.

B

Give me my drink is enjoyment plug-in right, yeah.

C

So it just shows you all the benchmark, it visualizes it perfectly so we're just running one single tag to put was just of one single test. But if you have multiple reports, you can compare them all.

C

Now this is what I was talking about the jetty leaving threads running. So what finally happens that gem, it shuts it down. So this is the second folk that's running I.

C

Just also should know how to just create a small benchmark right so.

D

C

That's all you have to do for creating a benchmark and.

B

How do you configure a number of iterations and other things which could execute it? Yeah.

C

So they're here in the benchmark, runner, so I guess I'd be using average time right now and we can just configure the number of iterations. So we have two threads to folks and to warm up iterations.

C

Now I think the benchmark is completed, we're just waiting for it to terminate.

C

Yeah so we got a, we got a fresh resolver.

C

So our score was zero point, eight three microseconds for one run of the benchmark. That's the average time.

C

So these are the results this time.

B

C

Is the confidence interval this black bar.

A

B

So an equation of squares.

B

Yeah, if you have any questions, just feel free to ask them in the chat Oh in the just unmute yourself and ask if you want yeah. Maybe it makes sense to talk a bit about the next steps, because yet currently we just started- and our current one as proposed in the emergent list- is to actually be a part of jenkins dishonors, so that it would be used by default oddity. It can be used from Feigenbaum, so easiiy it will be created, make it available to all links, plug-in developers using standard tools and yeah.

B

If you have any feedback about that, please let us know: okay,.

B

So if there is no feedback right now, you can probably appreciate this project simcha. Oh it doesn't, but you want to say something.

B

So yeah I think we can just think up on the current activities.

B

Yeah, if you want to be taken just share my screen solitaire, we can take a look.

D

Do you see it yes,.

B

Yeah, by the way, this is how each other. So here all the discussion happens in the hall strategy plug-in and by now we already have first through Jesus changes integrated sweetie if you tickle the releases so maybe yeah. Two days ago, there was a new release of strategy playing.

B

There is a lot of changes already indicated for performance testing, so right now for most distance is disabled and on Jenkins file, because we need to speak to the infrastructure team about how they properly implemented, and you have O'leary on the course or maybe it's a good opportunity to discuss that yeah.

B

He also good Jake asked support and we also got some bug fixes much changes to come soon and yeah. Maybe we could really use the opportunity so yeah what was originally done? He did performance such as run as a part of Python steps. Okay, just invoke and that built again and then we just use another, not acquisition of it from benchmarks fees. The benchmark for so this is a fourth age. Now enables gmh immediate profile, but yeah. The question was: what would be the best agent types to do?

B

Such performance benchmark, and so only if you have some suggestions about it, it will be much appreciated.

B

And sure maybe he's not oh, he cannot talk right now, but yeah. Then we can just yeah. Then we can adjust the take it offline and yet discussing this Jenkins developer man increased thread, yep Thank, You, olivine, so yeah you see, might shut up ops and yeah.

B

Everything is visible, so we have no way to meet goal, for the project would be to would enable these components. But here right now gave me just wait for feedback and they gave you one go to Sudan because in the rest.

C

Yeah so I did add a few more pull requests for adding more performance tests.

B

So here is free ones, rated yeah.

D

B

Holder benchmark is one of the benchmarks which has mitigated to the community feedback. So yeah we've had some discussion about use cases to check and hear this use kisses implemented right now and if somebody has any specific use cases for whole strategy usage but feel free to provide this feedback, because yet probably me good Justin human mesh marks or them and it will be a base for the next codon iteration so that we can improve performance there.

B

And yeah I guess you need reviews for all of these requests right.

B

Because regulations, the repository can do reviews and yeah. My plan is to make review board so that it will be triggering I.

C

Would like to discuss about the times that these benchmarks are taking, because they seem to be very small.

B

Yes, so I guess is somehow related to this pull request. A transformation for more favorite configurations is that right.

C

Now, what's happening is that the time taken is too short, mm-hmm.

B

B

Yeah, maybe you could show your screen and show you know: oh yeah I can living it. You can you need to see me too so.

C

Yeah, if we just take a look at the previous results, I'll share my screen. Okay,.

C

This one it takes 0.8 of a microsecond to do one iteration I mean this may not be the most complicated benchmark that we have burden.

B

So what would be your preference, so yeah? One of the ways is to actually create a test which involves small technician, checks, or maybe we could add some artificial artists, for example rendering of web UI, because yeah one of the stories of EDA implement is a part of application. He created an instance really represented slow down off jenkins web.

B

You are a little strategy of like in this massive number of regular expressions, so what we could do even in benchmark, we could try rendering jenkins web yari, because in jenkins test harness that is the framework unit. Music it'd be huge. They can navigate to web joy as a part of the benchmark. So we needs topic to consider because, if you want to have free will heavy smoke test, maybe we can just amoled request to web UI there and make jenkins to render that stuff.

B

It uses fun, solid areas and other such things so yeah it only it's the performance pretty well I mean I believe that it could provide you pretty long code execution time for the bench. Okay, I'll try that yeah. Actually, it would be a good experiment to try. It out is performance testing framework because yeah so HTML being it shouldn't, be using Jenkins fall, but here there may be some interaction.

B

So here, if you want I, can just show you how it looks like.

D

I'll show you my screen.

B

B

Whatever so here we have police to test and I believe.

B

There are some artists: hey. You may see that web client has actually initiated our client. Jinkies wrote great word friend, so you may see the problems start here, because here what client is actually created by Jenkins does harm us. So we, if we take a look at the code.

B

So we have, it comes from the wrecker.

D

I didn't think it's too.

B

Complex but basically just return a new, a plan.

B

In that class and this not cross switch the interrupts with Jenkins context, so let me see that it takes so some information like that. Basically, it's a client so frankly think and I believe that they eating shrugs is upper class, so we'll set up URLs and other such things. Most likely. This code can be moved to the performance testing framework so that we can implement web client requests from the performance test framework.

B

When so, basically, what you could do it would fit web client and then a in your instance by Jake, ask if they've created multiple use with different set of layouts. So, for example, we had the discussion of all of you this morning in the chat, so we could have for this. All of you and the number of other views, and that may become, can have an Amish much it's just access to different views. So we could have a good set or smoke test configurations.

A

Does it make sense.

D

B

So it definitely wasn't in the additional scope.

C

Your voice is not cleared anymore. Sorry.

B

For this piece, it's something.

B

We ahead of schedule right now, so if you want to experiment with such more heavy tests, I think it's a good time to do so.

B

B

Yeah so, basically, maybe create a separate task for what you are invest test and yeah. You can do it as a part of this sprint.

B

Okay, so basically we have a recording called it. So you can take a look at the implementation and maybe take some examples. But here web client is pretty straightforward until you start a person or device and I believe that in your case you don't really need to parse web UIs, you just need to retrieve them, so it should be pretty trivial.

B

B

So do you have any other.

C

Questions so any topics you would like to discuss so should I start extracting the framework to somewhere else.

B

Or maybe it's something to discuss at this poll, so we have several Jenkins contributors and my question to all of you guys. What would you say if we had it as a port of genius test harness so just offered out of the box? What do you think.

B

Markey Rick.

D

I'm here I think a deep of functionality would be really good.

C

D

I see so far in what I'm looking at.

B

So we're having default. Fortunately, we basically have two ways to implement that, so one of the ways is to just make it a part of jenkees test harness directly. Now the parts have a separate repository, for example, in Jenkins we have our own implementation for docker based testing. It's dr. fixtures, just a second Jenkins yeah, but our main problem is the final. Well, it's not really a problem at that.

B

We actually have a series developed with the Jenkins, probably imaging vistas harness so by putting in direct inject as promised, we could have a code without reusing that, so we can just have two implementations aligned, which would definitely help all contributors who want to adopt the framework because they would have a similar set of features like engines through we I could see. I could see benefit in that.

B

My preference would be touch if you are fine with that. We can start working on this topic so just by suggesting couple of questions test harness which takes the framework for today, so I mean Java closes and other things we will need to think about about the packaging. So usually we put the information. All there is a number of projects, so my suggestion would be to have something: kinda Jenkins, for example, Jenkins /, benchmark Inc, or something like that, so that you put good to the newer bars because he had the most of code.

B

He is actually yeah. It actually comes from hot sometimes, but it really makes sense to use Jenkins package for the new code base and yeah. Once it's done once the new jinkies test harness is released. What we might need to do is to update so plug in for our framework, which is the full complementation, and here.

B

Is the offer directory is a part of plugging bomb, so once the drinks, the prominently the past is released, you can suggest a patch here and it's also possible to add profiles.

B

Do pretty much the same for performance test so, like you didn't at all strategic like you can just add a profile for benchmarks and I, keep it there for now, so that anybody who is interested you have a profile. We have some documentation in the repository and it would be a good step. And then, if you want, you can defy pattern library to use it by default and some conditions.

B

So this is the default we were discussing before, but him first steps like kinky stuff on us. If you started doing that, I think it's a good time to start doing that, because I believe we can remove consensus, but I did I, do not expect really issues and about the including Gmail directly. In touching his harness is not.

B

Swim and this story is also something we can proceed with.

B

And do we have any other blocked stories right now, so you mentioned this budget issues with JT and the issues with Jeff 200 and it's jab 200 issue may be a problem or once we start doing Christa Klages, because he arrests, API just really depends on this. Just come by Aisha.

B

Are they looking through you right now.

C

Not with the current implementation, if we share code with Jenkins that furnace, that does it by default, so that would I mean contradict with the configuration escort part I.

B

B

We can just proceed and yeah thanks either for the demo. It's a really good progress. We couldn't just started this week, but you already have something to show and yeah thanks a lot for the in bed.

B

Okay, if there is no other topics, thanks everybody for participating yeah, this video was recorded, so we will get possible to you on YouTube and if you have any questions, just ask in the inkeater shot so that we can discuss them there. You.

D

Have a good day, everybody.

D

Right thanks: everyone, okay I'm.