Jenkins Google Summer of Code Office Hours, 7 Mar 2023

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: GSoC 2023 Office Hours - March 7, 2023

Description

Special project idea presentations:
- Exponential backoff and jitter for agent reconnections
- GitLab Plugin Modernization

A

Recording is on and uh wish everybody a good morning, good afternoon or good night around the world. uh This is a special edition for gsoc uh office hour, where we will uh answer questions solve doubts about two project ideas for the Jenkins Google summer of code, um and we have several mentors here on the on the call.

A

So just a brief presentation for myself: I am John Mark Mason I'm, located in Brussels, Belgium Europe and together with Alyssa, who is on the west coast of the United States, we're both org admins, and we have also Bruno Bruno, who is located in Northern or France. So these are the organizers and we have um mentors around the table. So we work first vessel. Can you just present yourself who you are where you're located.

B

Sir I'm Bethel and I'm located in Northern California.

A

C

I'm I'm Mark and I'm, located in Colorado in the United States, not far from the Rocky Mountains.

A

Great so morning, for you too and Logie, where are who are you? Where are you located.

D

Yeah hi I'm logi I'm, located in Sri Lanka.

A

I I should remember it yeah each time so and how do you pronounce your first name: Logie Logie yeah, okay, Nido training, for that. So welcome.

A

um So without further delay, we're going to uh discuss the first project idea, which has quite an impressive uh name so exponential back off in Jitter for agent, uh reconnections uh I, think this is a subject where basil has good knowledge about. He proposed this project idea. So, first of all, could you present the project? What it is why it's important and then we'll open up for a question for housekeeping the presentation of the project should not exceed 10 minutes. Sir sir,.

B

um Well, as the uh as a user of the Swarm plug-in, there's been multiple pull requests over the years to solve this problem, which is that when we starting a controller, uh all the agents we conduct at the same time- and this can be scalability limitation if too many agents are connecting to the controller simultaneously.

B

The controller could reject some of these connections, and essentially that means that the restart of the controller would uh would be only partially successful if H, if some of the agents were not able to reconnect.

B

um So people have uh reported this with both the Swarm uh agents, as well as the standard, remoting agents. uh So swarm is effectively a wrapper round remoting and the uh um the problem is basically to.

B

This is basically a scalability problem right um and the idea behind this solution would be to introduce Jitter so that these agents aren't all trying to reconnect. At the same time, um the main challenges for this project are uh number one, there's two different implementations of this, so the code is not already in a clean, State, there's essentially duplicate logic in standard remoting framework as well as in swarm.

B

So um so that's already going to be a challenge, then the other challenge with this project is demonstrating that this Jitter is actually having the desired effect um on a busy controller. So you know it's one thing to Implement an algorithm, that's defined on paper, but the other question is: is this actually beneficial to users and the only way to really determine that is to do some kind of scalability testing and to see okay?

B

Well, how can we reproduce this scenario where the controller is being overwhelmed with agent reconnections and once we can we produce that scenario? Is this edition of Jitter actually making a difference? That is? Is it improving the success rate of reconnections? So that's really this project. In a nutshell,.

A

So we we have already just for for my general culture, so does the remoting part have already the um exponential back off? That means that every time an additional time is added to the retry, does it already have that I.

B

Don't remember, I think, there's some there's, definitely some stuff missing from remoting that is present in the Swarm version and and vice versa. Maybe in other words, these two implementations are not a feature parity with each other. So, okay, that's part of this project is to get these two different implementations up to feature parity and I. Can't I can't remember what remoting is missing. It's definitely missing something right.

A

Mark uh raised his hand.

C

Yeah basil, could you could you describe for those who might not be familiar with what the term Jitter means in this case, give a little more a little more depth on it. I had to go. Do some some looking to see? Okay, what's Jitter in this in the context Etc yeah.

B

Jitter in this context means adding a randomized time delay uh to spread out uh to spread out load so um in the project proposal, there's a link to an AWS blog post that has a nice diagram of this, um which kind of shows that uh adding Jitter will kind of increase the.

B

um What's the best way to describe this it'll, any Jitter will uh will ensure that there aren't these big Columns of uh dots in the same time span and if, if you uh I mean I, just just go, read the AWS blog post, but they have a diagram of this where they show how essentially adding sugar distributes the load in a different way, so that uh so that the server isn't overwhelmed within any particular time period.

C

Okay, so it's just Jitter is a random random, startup delay to the initial connection and that that by doing Randomness, when a thousand things connect, we don't get a thousand of them in the same exact. Second right, the idea is spread it around. Okay. Thank you. Thank.

B

You sorry I should have read the paper. No sorry, it's spreading it around within a limited boundary.

A

With a limited or unlimited, a limited boundary right say between so and so many seconds choose where you're going to start.

B

A

Good, are we trying.

B

Is a retrying is a deceivingly, complex topic, um because, if you, if you look at um there's like a one of my favorite, libraries for retrying, is tenacity which is a python Library and they have a really good documentation. But if you look at the tenacity documentation, you can see that there's all these different ways that you can retry things and there's not.

B

Some of them are better in some use cases than others, but it's not obvious to anyone. Who's who's neuter to retrying that there's this much complexity behind the topic. But you know, for example, you can uh you? Can sleep fix the meta time between tries you? Can uh you can increase the amount of sleeping between retries, which is um called exponential back off, which is done by TCP? You can add Jitter, there's all these things that you can do and they make sense in some use cases, but not others.

B

So the use case that we're talking about is the Thundering Herd problem, and so that that is the case that we're discussing with controllers being flooded with agent reconnections and uh in in that use case. The identified solution have industry-wide is to use exponential back off and Jitter. So that's that's the medicine for this problem. It's.

A

A medicine, um I I think one of the important Dimension about that is uh to demonstrate that it has the desired effect, so the the the testing so the usual testing, which is unitest integration testing. But here we really need to do an end-to-end test of that feature. Yeah.

B

The testing is probably the hardest part of this. Actually because I mean it's, it's relatively straightforward to refactor duplicate code and to add a loop that goes, and you know, maybe not trivial, but uh it's something as well conceptually. It's not uh there's nothing complicated about adding a for Loop. You know to some code um in practice. It's going to be hard because this code is a mess and it's it's extremely messy code so and then, like I, said, there's two versions of it as well.

B

So if, in practice it's like one of the most difficult for Loops that you're going to be addict, but but you know the testing is really the hard part here, because uh how do you really uh demonstrate that this is making a difference?

B

um You know uh the there's: it's not easy to set up these kind of scalability environments, even if you have a lot of computers, but especially if you don't have a lot of computers.

B

um If you're dealing with a small number of machines- maybe even one machine um you know it would be- would be fairly challenging to kind of replicate this scalability limitation, where you're seeing agents being dropped um so that that that's really a difficult part of this and I'm.

A

Not sure yeah sort of in your previous life, you experience these heard effects.

B

A

B

A

E

B

Running a big Jenkins uh deployment at a previous employer where I experienced this firsthand and I've seen other users, um people have submitted the pull requests to swarm, to add Jitter and and uh I've even seen uh social media posts from uh Jenkins users. Talking about this problem, so you know one idea might be to try to you know, connect with one of these users and try to get them to deploy. You know a test version or something like that, but.

E

B

You know that that isn't certain that we'd be able to find someone like that or that they'd be willing to uh to deploy to their production environment. Some of this code and.

A

What was the the number of Agents we were talking about? uh Are we talking about 10 hundred thousand well.

B

My case, it was hundreds uh I'm, not sure what um I'm not sure what the Netflix case is I think it's hundreds to thousands, and there were also some people in the Swarm plug-in filing pull requests and I tried to ask them for more information and I never got any so and.

A

Excuse my curiosity, but and what is the the effect of the herds? uh The.

B

Yeah, uh the fact is, if an agent can't reconnect, then that job is still running on the agent, but the controller has lost communication. So uh when the controller is coming back up after the restart, uh you won't be able to get any status updates until it reconnects with the agent.

B

So um you know for I think I'm, not sure if remoting does retries at all right now, which is I, think the scary part I, think remoting by itself might might not be doing retries, I, don't I, don't know, I, don't really use remoting directly because I like I like swarm personally, but um it's it's not really the standard way of doing it. Swarm definitely does retrying, which is you know, critical in this kind of scenario.

B

But then again you know if you're, if all the agents are retrying at the exact same time, then that could create a worse problem, which is that the controller just gets completely hammered with connections at the same instant. You know every 10 seconds or every 30 seconds, which is why that, which is why Jitter and backup are both needed um but yeah.

B

Basically, the effect is that the uh that the controller could be made inaccessible if it's receiving so many requests at the same time or the jobs wouldn't show status updates until that reconnection takes place. Okay,.

A

Last question: before opening the questions, I'm keeping the clock in in my.

E

A

uh This project can be huge, so there is a lot of a lot of work possibilities. There are a lot of uh holes. You can start jumping rabbit holes that you can go. uh Can it be cut in in different phases or different and and say have ideas for another edition of Google summer code or or is? Do you see a strategy like that.

B

Yeah absolutely I mean there's, there's no harm at all. It's just simply consolidating the duplicate code that exists today and not included and not having any regressions right. That is a change that could easily be accepted without controversy. If the the existing code is deduplicated into some sort of common uh function or common class or whatever the challenge there is uh that a command line interface needs to be added to express some of these ideas right to to express the the back off period or the amount of time between tries.

B

The number of tries uh Etc so uh in designing clis is always tricky, because it's effectively a contract that you can't break once you once people start using these command line options, so um so a lot of effort needs to be, in other words, there's this challenge, where even if you're, only implementing a subset of this, you need to design the CLI for the whole thing before you can Implement a small part of it, because designing a CLI is one of those things where um it's effectively an API that people commit to.

B

So you can't you can't just do it in incrementally. You've got to design the whole API, even if you're only implementing part of that API up front. So I think that's, that's! That's the only that's the biggest impediment to Breaking this up into incremental tasks is that the CLI really needs to be thought out this one completely before anything is implemented, even if, even if the implementation is only partial, it's easy.

B

It's easy to leave out, see a like commands or sub commands, but it's much harder after you've already implemented a CLI to go back and say. ah Well, you know. Actually this I need to rename this argument or I need to add some some uh mandatory Flags to this other. You know it's a lot harder to do that.

A

There is also in other interesting work package is being the how to test it, and there might be some thoughts, designing and tooling here. I will stop this presentation. Part I would just would like to give the word to to log d.

A

uh If you have additional things, you would like to share or questions or or points, don't forget to unmute you yeah.

D

So I would add upon that some points, so this project is very vast, so the remoting module is very integrated. There are a lot of dependencies at the plugin as well, so the they'll be very challenging task that when we try to uh push this code changes to this Upstream repository. So when it comes to uh the production level of this uh implementations, then we need a very hard testing for this, uh and students can do their own research because this kind of uh like we can.

D

We can use the exponential for now, like that's a very uh industrial standard for this uh retry mechanism, even in my company xyzatz or we are using the exponential rate rifle even for the micro service intercommunications.

D

So exponential is a good uh choice, but students can do their research on research and they can propose what what whatever it works. We can uh Implement that yeah, but it's a very uh vast project and and also interesting projects. That's why I joined some enter.

A

Great tool to take the analogy again: a big mountain okay who wanted to jump in?

A

Oh okay, so it was not effect from from my side, open up the question to the rest of the audience: uh nethra, Bandit or.

F

uh Hello, everybody uh I I, have some questions related to uh how the unification of Jenkins, remoting and swample again is to be implemented like currently remoting waits for 10 seconds after every after every attempt to connect to the controller. Currently it waits for 10 seconds uh so, and the Swarm plugin has three: has a linear back of linear, retry, algorithm and exponential back of exponential retry algorithm. So do we have and it all, and so on? Plugin already has the CLI function uh where we, where we can override the default exponential back of algorithm.

F

So first do we have to uh do. We have to implement the things that are already in swamp plugin into the remoting module and then unify it.

B

Yeah, that's exactly right and and the Swarm CLI is missing. Jitter yeah.

F

E

B

Together, the first step would be to design and I I would not consider the Swarm CLI to be a work of art to be emulated. It's evolved haphazardly in the standard for introducing a CLI change to remoting is a lot higher than the standard for or I'd say that yeah swarm has just evolved in a very messy way over the years, but, um like the main, the main challenge is coming up with a good.

B

A good, CLI and I actually wouldn't be afraid to break swarm consumers because I don't think too many of them are using this or even if they are, it could be announced. You know ahead of time, but the key is to design a CLI that does make sense and is easy to use. That includes all those features that swarm has and Jitter, which swarm does not yet have and to really find a way to express that to the use, because I think right now in swarm the exponential backup, it's a very strange CLI.

B

It's it's not an intuitive uh way of expressing the arguments and the the the intervals, it's basically just there as a historical accident. So that's where I would start is uh really figuring out. How do we express these options in an intuitive way and then from there? How do we retrofit these existing implementations to align with that intuitive CLI, which would only exist in in theory and then in practice? We can slowly evolve the existing implementations to fit it's going to be easier for remoting, because there isn't any support for this.

B

So it's a clean slate, but that's both good and bad in the sense that it only gives us one chance to get CLI right and then for swarm, there's a little bit of so you know it's warm if we need to change the CLI to really fit this ideal, that we could do something like have a transition period or something where these old options keep working, but we deprecate them and try to move to the the options that we think make sense, so designing a good CLI designing a an intuitive CLI that can be used by a lot of people without making them think too hard about what options they need.

B

I. Think that's! A big part of this.

A

B

A

Benson, are there other questions or.

F

Yeah yeah I have some more questions. Sorry for taking so much time. Oh.

A

F

F

My question is about the stress, testing framework, uh I'll search the web and I have found about uh Apache geometry like we can use that to uh mock something like that where uh at 10 or 20 age we can. We can mock 10 or 20 agents using that I I got I got to read that from some blogs on the web, but it's not the optimal solution, but it's it's something that we can use to for stress, testing on remoting and the CLI part I I.

F

Think, because it is a one chance to create the CLI if I. If right now, I try to uh use remoting from the CLI it, it says you are using it from the console, use it interactively So. Currently it has something like of a CLI implemented, but but it does not work, it just says: use it interactively.

B

F

uh So like does the remote already have it because the com, the code, is really complex. Just like you said that is something I got from the high level overview.

B

A

So uh when did did you have a specific question so I think you, you raised a few points about uh the the testing framework, so you explored a few ideas.

F

A

F

A

And having read your your initial uh draft, so I think there is is an interesting topic to to research, although there's choices that will have to be made uh one of the things that you can start and I'm I'm talking to the audience here is to Define so to to know how do we know that we have the problem right. So this is the first question and the first research item that can be part of gsoc, so you don't need to have it but State the problem uh there. So how can you say there?

A

We are there. We are now we have the problem and then the second part is to think on. How can I increase the load or create a load where I can demonstrate that the problem is caused by this retry a load and- and this would be already I believe looking at at bezel uh already one big step forward in starting to solve the problem and and then uh the the other steps uh are so it I, I, think seeing from the reaction from bezel I think I'm I'm on on Solid Ground.

A

There say now: I need to to look at the clock. Yeah.

B

The other question to be accepted to be accepted. Ultimately, our bar for accept uh well by accepted I, don't mean uh from a gsoc point of view, but for for any code change to remoting to be merged our standard. Would our standard of approval would be for it to be demonstrated to solve some sort of problem and to not cause our questions and the demons.

B

So the demonstration that it's solving some sort of problem would either come from the results of a stress test framework or from an end user who's experiencing the problem, trying out the build and Reporting success and that's a lot harder because end users are they're not very good at trying out beta code from us, and even if, even if they're willing to they are not always they're, not always good about reporting back the results or sometimes their results are questionable, and you wonder if they really deployed it correctly, et cetera Etc.

B

So when you're, not the one doing the testing yourself, it's a lot harder to convince yourself that the change has had the desired effect. It's a lot easier if you're also the one doing the testing. So that's why I'm really hesitant to rely on other community members to do testing, because it's very hit or miss with them, sometimes they're, very responsive and test your build right away and other times other times. It just hits a dead end. So yeah.

A

I mean first first to.

B

Accept uh PR to remoting we'd want to see some evidence that it's having a beneficial effect and a real world use case, not just a not just a mock test. So um whether.

A

That's whether.

B

It's a local test or an end user. Like I said um we generally wouldn't accept a pull request. Unless we can, we can tell that this is better than before, and it's not causing a regression yeah.

A

So I read from that that starting with test and digging into the test framework is a a first good step for that project. Yeah.

B

And uh you know like Loki mentioned you know, maybe uh Jitter isn't uh the best solution to the problem. Yeah or maybe there are other solutions that you could experiment with and choose which one has the best effects if you've got a way to increase load like that, so I.

A

I, like that uh here, I'm I'm, sorry we'll we'll pause. This discussion, uh we'll try to keep 10 minutes at the end, will not exceed the hour promised, uh because these are interesting questions and interesting discussions, but uh I'll be so we'll keep 10 minutes at the end for uh questions uh I'd like to cover the second uh project, um so the gitlab plug-in modernization, um I think Mark. You had things to tell us or or basil, who would like to talk about that project. I.

C

I think basel's more credible, I'm happy to talk about it if you'd like, but let's.

B

Talk about it, do you want to introduce it at a high level and then I can talk about some of the challenges with Jersey and rest easy. Oh, yes,.

C

Yes, yes, oh and that's, okay, Basel has just noted a sort of awkward mistake, I made. So yes, let's highlight that first, so at the high level, the gitlab plugin is important to Jenkins consumers who are using gitlab as their SCM system. It allows them to do all sorts of interesting and useful things updates of of merge, request comments, status, changes, Etc, and so the gitlab plugin is quite useful.

C

However, the gitlab plugin is outdated in many of the things that it does and much of its implementation, and there are specific things that need to be improved in that implementation. Now it's important that we have a strong integration with gitlab, just like we have a strong integration with GitHub and bitbucket SCM providers matter deeply to Jenkins users. So we want the gitlab plugin to be very strong and right now it's it's not had an active maintainer for a while.

C

It's not had if Chris Stern just barely started on it again and we're we're deeply grateful for that. But there's a lot to be done to modernize. This thing, I think that's all I wanted to say Basel. Do you want to take the next step, which is more more relevant technical detail.

B

Yeah, the gitlab plugin is communicating with a given gitlab server through its rest API. So gitlab has a rest, API I think it's I can't remember if it's Json based I think it's json-based um and essentially anything that's communicating with. It goes through that API, including the gitlab Jenkins plugin, and we've had this fairly constant thorn in our side for the last few years. That I've noticed, um which is the way that the Jenkins git lab, plug-in communicates through this rest.

B

Api, it's using a very old library called rest easy, and not only that, but it's also using a very old version of it. Yeah and this this library has been a bit of a challenge to maintain it.

B

It's as far as I know, it's the only thing in Jenkins land in the Jenkins ecosystem that is using this particular rest, easy Library and this library is not very easy to work with, but it's also just very fragile and this there's something to be gained by being on a more common path with the rest of the universe right, we're effectively.

B

I, don't know if we're the only people in the world using rest easy to communicate with gitlab, but we very well may be the only people on the planet doing it this particular way and as a result, we're subject to a lot of regressions and problems. That simply don't happen to other people, because other people are using a more common path that we're that we're not on.

B

um In particular, the the standard way of communicating with gitlab from java is to use the um there's a there's, an official or semi-official Java, Library um I forget what it's called now but um I'll. Look it up, uh gitlab 4J, um so that that's a very nice library that um gives you these Java classes and you can invoke methods on them and it will figure out in the back end how to translate that Java method call into uh a Json request for you, but that that gitlab 4J Library is that's.

B

That's the common path that the rest of the universe is using to communicate with gitlab. We we've actually already bundled it as a Jenkins, Library plugin, and it's being used by the gitlab branch, Source plugin for Jenkins. It's being used by the gitlab authentication plugin for Jenkins, um so this this is kind of the standard code path. I would like to follow uniformly um in in all Jenkins plugins, including the the git lab Jenkins plugin.

B

So the the idea behind this project would be to migrate callers from uh rest, easy to the git lab 4J library, and what that would accomplish would be that we'd be following a more standard deployment. Paradigm we likely wouldn't run into all of the regressions and bugs that we encounter with rest easy. We just encountered a new one this week. In fact.

B

um So uh that's really. The project, in a nutshell, is bringing us into a more standard deployment pattern by eliminating the use of a very old library in favor of the more commonly used gitlab for J Library.

C

So so the Buzzle buzzles comment changing from one library to another. My brain shows big signs that say: there's probably work involved there and it's we don't want to regress. So there's testing involved, there's familiarization with hey. How do you use this thing? There's probably interactive exploration needed very likely that additional automated tests will need to be added in doing this because probably discover oh whoops, here's a piece of code that called rest easy had no interactive or no automated tests, those kinds of things and major service to the Jenkins Community doing it.

A

I can feel that so interesting work, useful work and substantial work so.

B

A

B

Turns out this week we just uh we just upgraded Jersey and Jersey broke compatibility with rest, easy, so I filed this now an issue filed with Jersey um to restore rest, but I could tell that the person who is responding to the issue was like. Oh, this is very interesting. You're using jersey with West, easy and I think they used. The phrase very interesting and I was like in the British.

A

A

C

Right, interesting in the sense of completely dumbfounded, what were you thinking like.

B

Why are you still? Why are you still doing this right? Exactly exactly yeah I can Envision yeah. This is uh this is just it's just a thorn in our side for a long time um this it made sense I assume when this was originally developed, but it's just no longer a viable library to use good.

A

Okay, are there uh questions uh from the audience about.

G

um So I have a question.

A

G

It can be a silly question, but I just wanted to know what are these Corner cases which should be handled uh doing this I didn't understand those uh authenticated proxy servers, unauthenticated proxy service, all those Corner cases which this project is talking about.

B

Yeah yeah any kind of any kind of library that does http calls is deceivingly simple, because you might think, oh well, you know I'm just calling a rest API from an HTTP client and that's usually very straightforward, but there's all there's always tricky cases that go along with HTTP clients, for example, proxy servers.

B

A lot of companies that use Jenkins require all communication to go through a proxy server, and what that means is that the HTTP client Library needs to be configured to use the proxy if it's been defined by the user. So we have, we have a place in the Jenkins UI, where you can specify a proxy server and from there that proxy server needs to be plumbed through into in the in each Jenkins plugin to whatever libraries are performing HTTP requests, and it's not just it's not just the presence of a proxy server.

B

Some of these proxy servers also require Authentication um yeah and some of them. You know there's also like it gets even more complicated because there's um https proxies that have TLS certificates that need to be valid, so it can. It can become fairly challenging to test some of these cases and you usually don't find out about it and unless you break it and then some past, some user of Jenkins is like oh well, I use, Jenkins with a TLS proxy with uh ntlm authentication and that stopped working when I upgraded we've.

A

Been we've been there, I know the one who invented corporate proxies should burn in hell, because it's been a nightmare. I got a lot of gray hairs because of that and Bruno lost his.

B

Yeah because we already know proactively that without any explicit testing that would probably break so, there's a need to do explicit testing. If some of these scenarios- and you could find you know, Docker images of these proxy servers- I've I've tried a few of them.

B

None of them are particularly great, but if you know there's ways that you can set up a local proxy and test communication with it and and such, but it's it's usually an extra step of testing that would not be intuitive to someone who's just using this using an HTTP client in in the usual way.

B

um It's always it's always an extra level of thinking and testing to support this.

A

This front did that answer your question.

G

Yes, yeah somewhat uh I, guess I'll understand it better when I start actually understanding the project in people's way. You.

A

Have other questions yeah.

G

So it's related to The, Proposal Part, uh certainly Proposal Part. How how do I uh elaborate these steps involved in this project, like uh uh so like how? How do I explain it.

A

um There so did you already pull the proposal template. Did you look at it? Yes, there are some some guidance there and so.

G

A

Beginning you expose so you do summary, and then you expose what is the problem that you're going to solve with that project and then you're going to describe uh the strategy so what you're going you're going to do uh and and so and this you can use whatever technique? You want uh to explain.

E

A

Drawings uh some steps, so there are various ways and you can try them out and you submit your document and say: do people I'm, I'm a very good guinea pig for that uh because I don't know the details of the projects. If I don't understand it quickly, uh then others will will have uh problems. So, okay, try to explain it to a TR. The good way is you have a a friend at University or so, and you have just 20 minutes to explain to him what you're going to do.

G

A

And just use use that, and there is also a part where you're going to explain how you're going to split your time- okay during the summer, so I'm going to do first that I'm going to do that, and this will allow us to compare with other proposal and say. Okay, this person has a sensible plan, has understood the problem. Okay,.

G

A

Starts do a first draft. The first one is like going into cold water. It's it's! It's uneasy! It's not not fun, but start writing something. And then you ask people for their opinion. Does it.

G

Make sense: okay, yeah.

A

I know it's it's scary, but here we here to help you and and we we want you to understand, learn from the process. Yes,.

G

A

Did you understand the problem or are there other details, I understood what what uh Mark and basil explained so I'm even tempted to start looking into it which I shouldn't? But it's yes.

G

uh Yeah I have understood them or most important stops. uh The only part which is triggering me is a pre-testing part like uh should we like do tests, often uh to make sure the process is like going well integration test unit tests and all those stuff.

A

Yes, so believe an old man like me, and so what has not been tested will fail. So testing is important. Testing should be automatic, so check. Everything. Mark has also opinions on on that one. So I'll stop right.

C

After your, your, your in your comment to hey automated testing absolutely and run the run, the automated tests run them as a way to get started. Watch how they behave on your system, maybe run them in a debugger and watch how they, how they execute. Certainly look at the integration tests that are in that plug-in if there are and see how they operate.

C

Don't forget- and this is the crucial one that I tended to forget: I had a bias towards test Automation and I sometimes failed to do the interactive testing that would expose bugs that were very obvious to me and not valid I'm, very obvious from the user interface, but not checked by automation. So it's not just automated unit tests and automated integration tests, there's also an element where the contributor here must plan to do interactive exploration and testing interactively to be sure that it's well behaved.

B

So Mark the uh the plug-in has integration tests that use wire, Mock and wiremock is a library that effectively it it's a Bach http uh layer where it expects a certain HTTP request and we'll provide a canned response in lieu of a real gitlab server. That would be um they'll be doing real things in the in the back end. So some challenges, I, could foresee are the input to the to wire mock? Are you the expected requests?

B

Those might change. This might change slightly when you're switching from rest, easy to gitlab before J, without necessarily being wrong. It just might be like putting arguments in a different order or putting like new lines after each Json field, or something like that you know so. The the expected uh input to the tests might change, but not necessarily in a wrong way.

B

So the test might need to be updated just to adapt to that new style of input um and, like I said it's all, it's all mock based, so it really has no bearing on whether it's going to work in production or not um like I, said I, don't think wire mock is going to test anything to do with proxy servers or anything like that. That's all something that could only be tested interactively well.

C

And and whoever implemented the initial definition of those mocks, probably did it based on the gitlab version at that time and gitlab has certainly evolved right. Gitlab, the plugin.

B

Has existed, hopefully, thankfully gitlab has a pretty stable versions: API I'm, not sure what version we're using through rest, easy, but I think it's fairly recent. um That would be. That would be another thing to look into, though, is whether I actually don't remember whether gitlab4j is using the same version of the API as what we're currently using with rest, easy I think it's probably the same version, but they might be our rest.

B

Easy logic might be using an older version, in which case, if that's true, then upgrading to gitlab for Jay would not only be using a more modern library, but it would also be using a more modern, API endpoint of the gitlab server itself, and that could that could require the tests to be further. That would require even more updates to the tests. If they're, if they're mocking an older version of the API.

A

To ask for the other project testing understanding what we're testing all the test. Correct is the first important step to get yeah.

B

I tried to get I, wouldn't even bother trying to get the tests working initially. I I just go straight for testing on a real server and then because a lot of the challenges in getting the automated test, Suite working are going to be merely technical, rather than actually going like fixing real bugs like.

B

um Ideally, your test Suite is going to help you find real bugs, but in this case it's going to be mostly a hindrance. The test Suite is going to be mostly just more code that needs to be adapted to this new library, rather than a signal to whether your production code works or not. Unfortunately, so.

C

So I think that's safe to assume, then that it's okay to delete tests, if you find that it's not helping you that there isn't any.

B

Religious or or okay, somewhat, okay, I mean, but also, uh even even when they're only mocks, they are still useful in terms of exercising the Java code. So, ideally, we wouldn't delete them unless we have, unless it's primitively difficult to adapt. The mock.

A

Right: okay, good! That's right! Do you have other questions? You seem very interested about that. Go ahead.

G

uh Okay, so uh as uh discussed before so that, if there is any version difference between rest API and uh the uh gitlab 4J API, then uh it it would take more longer than the expected uh that's what they meant right.

B

Yeah there would be more more mocks that need to be adapted if, if the, uh if the existing mocks are using, let's say gitlab, API, V3 or something and gitlab4j is using gitlab API V4, then that would require more changes to the mocks. I mean that's a very easy question answer. We could just go and look at it. Look at the code and see.

G

Yeah, okay, yeah I guess so that's up here.

A

We have eight minutes to go. uh Somebody who didn't speak yet is a Smith if I pronounce it correctly. Do you have questions or.

E

um Hi hi everyone, no I, don't have right now, like I um I'm interested in uh exponential back of my Jitter project. uh I started looking, but uh I have not yet uh researched a lot because I have a job simultaneously. So we've got much time so yeah I will pick up the piece and uh okay.

A

So if you have a question or doubt in the next four minutes or five minutes, you can't shoot otherwise. If somebody else has question.

F

uh I have another question: uh maybe the last one uh I wanted to confirm the order of things that should be done for the exponential backup and Jitter project. First I should I should I work for, should I make the CLI, then stress testing framework and then exponential backup and Jitter in remote plugin. Should this be the the Order of Things.

B

The travels with that order of things is that, uh without a way to demonstrate the effectiveness of the change, it would be unlikely that it would be. uh It would be unlikely that it would be a pull that a pull request to remoting would be merged.

B

um If it's, if that pull, if a pull request to remoting is adding a Jitter feature that has not been demonstrated to make any difference and it's creating new API surface area, then it would be unlikely to to be merged, so uh I mean it would be. It would be fine to merge a remoting pull request that just uh that just unifies the CLI with swarm, let's say, um but in order in order for that to be designed, the entire CLI needs to be thought through, even if only part if it is implemented.

B

um So yeah order is very tricky. With this project, I mean ideally the ideally. The best way to start would be to find a way to reproduce this scalability problem and then to kind of iterate from there to find solutions to it and then from from there, even if, even if, uh even if no code change gets accepted into remoting, just having a way to reproduce, this scalability bottleneck would be a valuable contribution in enough itself.

A

All right, I have the same feeling here.

F

So the stress testing framework uh should do that, so that so I can demonstrate if the changes I am proposing uh will help and will help in making remoting better.

B

Yeah yeah and uh that I mean looking at uh looking at order of operations from a dependency point of view, a major design and testing dependencies having a way to reproduce this problem and to see if various Solutions are effective or not, and then from an implementation point of view. The first task that I, the first task from an order of operations, point of view is designing a CLI that is intuitive and meets people's expectations. I would I would not look at the Swarm. Cli is a good example.

B

That's more a good example of what not to do, um but even because, even if only part of that, CLI gets implemented. Even if, even if you only implement retries- and you don't do back off- you don't do Jitter, it's critical to design the CLI to it's critical to design this, the Jitter and back off parts of the CLI, because those arguments could affect the naming of the arguments for the parts that you are implementing.

B

So API design has to be holistic uh for it for an API to be intuitive, um so I would really start with I'll, probably start with some kind of performance, testing or stress testing framework or methodology from a design point of view and then from an implementation point of view. I'd start with the CLI design, even if even if only part of that design would get implemented.

E

A

We're three minutes from the end: uh if we have a short question you we can go ahead. Otherwise.

G

Go ahead uh can I just get a small uh sum up of like from where should I exactly start and like how should I go through the project? If a guidance like can be provided.

A

So we're talking about the git lab Improvement.

G

Yes, yeah gitlab plug-in modernization.

C

So if I'm in a venture Venture how I would approach it netra, so the improve a plug-in tutorial on jenkins.io guides on some things that you can specifically do to improve that plugin. Look through that that, through ape improve a plug-in look through that tutorial, see which of those could be applied to the gitlab plugin. If, yes, they've all already been applied, great.

G

C

G

Yeah uh we were talking about the uh schedule, build uh modernization right, which is our the tutorial.

C

Well, so that's what's mentioned in the video, but those same techniques can be applied to any plugin and the gitlab plugin I believe may even still have some remaining items in it that you could. You could begin a modernization using those small and simple steps.

B

There's no real, there's no real algorithm for how to implement a project like this. You've just got to rip out the old code and put in the new version, and it won't even compile right away and you have to get it to compile it's going to be kind of frustrating and then once it compiles it's not going to work, then you have to get it to work and that's not going to be.

B

That's also going to be frustrating and will require you basically debugging some library that you've never basically debugging some code that you didn't write in order to to integrate with it and then, once after all of this frustration, you think that you might, you might be done, but then you have to deal with proxy servers and authentication and all of these other insane.

A

You'll run away complicated.

E

B

Edge cases so I mean that's, that's essentially how the project would go. You know just rip out the old library now won't compile at all. uh Then you know add the new library still still not compiling, but you know slowly convert every method, call from the old library to the new and eventually after doing that, long enough, it'll compile but still not work and uh and then yeah, then from there get it to actually work in production, then get the tests to work. Tests probably need to be adapted.

B

The tests are going to be on, like I said earlier, they'd be almost useless because they won't actually help you catch problems in production, they're going to be more of a liability and that they need to be updated.

B

So that's I mean that's why this project is uh is something that has languished for so many years or months, because it's just not the most. uh It's just not the most fun thing to do, but unfortunately, software maintenance is full of tasks like this that are critical to the health.

B

The long-term health and projects like Jenkins that have been around for 15 years tend to accumulate a lot of these types of things, because the longer that your software is used, the more of the world around it is changing and your software needs to adapt I mean in the 1980s. You would never need to change software Because, the Internet didn't exist.

B

You could just keep it on some old library forever, but so now, now that your software is communicating with other software on the Internet, it's always got to evolve and adapt, and that's that's more true for 15 year old software, like Jenkins than it is for some newer projects. So it's part of the blessing and the curse of being an older and popular software project. So you end up with a lot of stuff like this. That needs to be adapted over time, but it can also be a rewarding Challenge and I.

B

You know I find this kind of work rewarding myself, because it's got a high impact on a lot of people, and it also teaches you how to be good at reading code, which is in many ways harder than writing code and it's a valuable skill to learn. um You know. I've I've definitely had a lot of benefit of my career from being able to read other people's code and understand it, and that's something you have to do a lot of in this project. So it's not all pain and laborious work.

B

It's there's also a lot to be learned and a lot of uh a valuable real world maintenance experience to be gained. So, okay,.

G

uh And one more like small doubt, this gitlab plugin has like around 57 000 installs. So does it anyway affect the project.

A

What do you mean with that? So you're go if you do a mistake that you're going to make.

B

A lot of people don't have a regression.

A

Now, here that's right, I'm going to tell you, don't be afraid of that! It's if I take the analogy of climbing a mountain if you're, if you're doing your work seriously, and you don't stretch too much or to take unnecessary risks and people are there to help you.

G

A

Will go well, so this is not something that should refrain you to tackle it. Okay, it's a steep hill and there are consequences, but they're, also big Rewards of that, because you're going to help a lot of people and and the coaching uh in this project will just make sure like basil stressed and saying you need to demonstrate that there are no, so no cue, but the project needs to demonstrate that there are no regressions, that it works as as advertised, and so these are secure.

A

Securities that you build you build, but don't be scared by the number of people. Mark go ahead and.

C

You're, you have a safety net, the lead mentor and the other mentors will do their best to give you feedback that helps you understand. If you're making a mistake now, I I have the fear you have and- and we should learn to live with that fear, because it's a good it's a good and motivating fear.

C

We shouldn't let it paralyze us, but rather we admit what I'm doing matters and it matters to 50 000 installations, and- and that is a great thing to matter to that many people and you've got the safety net of the mentors there to help you, but but you're your own best safety net right tests perform interactive tests, check that it behaves the way you expect use a debugger to watch. It run. To be sure, you know, what's going on, all those things are very healthy, healthy responses to the fear that you described yeah.

B

I mean the first thing I would do is Step through in a debugger the whole code path through rest, easy like from when it from when it's making its initial high level call.

B

You know going into rest easy going into gitlab, it's basically getting familiar with with what it's doing in the old version yeah, and then that way, when the new version doesn't work, you can start stepping through it and you'll have in your mind a pattern of what it should be doing, and then you can debug the new version and see where it's going off the rails, because even even after it compiles it's not going to work right away, it's gonna, it's gonna need some debugging but yeah.

B

Unfortunately, the automated tests are going to be more of a liability than than a help in this project. Right.

A

B

A

We're five minutes over budget, um so I'm sorry to have to interrupt because it's late in the Eastern hemisphere. uh Thank you for staying that that late. Thank you. Thank you very much for the mentors that joined the call and for taking the time- and it was a very, very interesting presentation for the um contributor candidates so start working on a proposal start to put something on paper start interacting.

A

You need to start somewhere and so that we can help you to uh to improve and uh and get ready, um ask questions on uh on guitar, Channel or better on community.jenkins.io.

A

And uh try to be specific uh in your questions. The very general questions tend to not catch attention of people so uh well, where should I start I want to do it, but what do I need to do? Oh, no make your homework, but I. Think the people here that that were around the table and I have done that so good. Thank you very much. Everybody have a nice rest of the day. Thank you very much for your involvement and don't forget. We have weekly office hours where you can join.

A

Ask your question exchange we'll try to have mentors regularly, also on the call I'll be there. So at least I can be the the relay for your questions. Thank you. Everybody bye-bye.

C

Thank you, Mark thank.

A