Jenkins Platform SIG, 5 Aug 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: 2020 08 05 GSoC Git Plugin Performance Project

Description

Jenkins Google Summer of Code project to improve git plugin performance.

A

I think we'll get delayed for the meetings.

B

A

That's okay, I'll: ask you separately yeah, so I'll share my screen now.

A

So um I think the first thing we can discuss is uh related to get tool chooser or yeah. We should discuss the progress with what uh with the get tool chooser so so we had an interesting conversation in the last meeting about about features that we provide to users which are not supported by jet, how to use how to get that information from jgit and how to um how to get that information to get tool chooser so that it can recommend the right implementation.

A

So ah so I'll just explain the problem. uh The problem here so that everyone's on the same page, so currently the user, while choosing an additional behavior, would not know uh if that feature is supported by jet or not. So the way it is designed is that, after for an example, if a user chooses a gate, lfs pull after that after that operation is run, the user will know that that operation is not supported through an exception and that is thrown um by jet.

A

So now the the challenge here is that we need to use this information. We need to get this information for the get tool chooser now. The biggest problem here is that git tool chooser is working before creating before the creation of the client and jget or cli get would be called when the client is created and we are using the operations.

A

That is how um it works. So so so, if we need that information, for example, if we need to know can we is the user um is get lfs uh pull uh supported or not. Now, if we need to know that before the creation of client, uh we had two options. I I thought about so I had a discussion with uh fran as well in the gita chat.

A

So uh so what I was saying there was that the problem is that jay gate the way, so it so that the checkout or any operation which is implemented by jkid, is implemented through certain commands like checkout command, or these are interfaces which provide the capabilities of different features provided in a checkout.

A

uh So so, when it implements it, it does not create any kind of uh a method to access the variables the field values like lfs pull, or so I need the field values to know what the user has chosen and then I can, if they are not null, I can say: okay, jk doesn't support that and we can throw an exception or do whatever we want to. uh We haven't hard coded anywhere that, okay, if we see get lfs pull somewhere, we say false or we say true.

A

It's always the way it is checked is that we we look for a non-null value. If we see a normal value, we throw an exception.

A

So there was no way for us to get that information out of jget with the current implementation, and so one of the ways I could do that was to change the uh the way checkout command is being implemented or clone command is being implemented within jkd api implementation, but that seemed like um so for me, what was easier and cleaner was to create a separate command which would not be dependent on any implementation because it because if the command depends on the implementation, which is jget or git, I would first have to create a client, and then I can get that information.

A

But I need the information before the creation of the client to do so. What I did was that I created a new command, which is not an interface, usually the all of the commands for the clone commands of the checkout or the push commands. They are uh implementations of a general interface called the git command. My command is not being implemented by any uh interface. It's uh is not it's not coming from any interface. It's it's a separate class uh which so what it does is it. It contains all the features.

A

I've aggregated all the I've collected all the features which are not supported by jgit. We can instantiate it before calling any client, and then we can use the decoration uh we usually do of commands from extensions so for any command. If we want to add the information which a user has provided to the command, we use decoration tools.

A

I've used something similar to decorate or, let's say, determine what other options user has chosen and, and then the unsupported command contains all of that information and it can determine if j get is to be used or not. It just provides a boolean that if jacket can be used or it cannot be used and then that boolean can be taken by the git tool chooser to modify its recommendation in fact not modify its recommendation. I have the I have changed the constructor now earlier.

A

The constructor only needed a gate, sem source object or a repo url to instantiate. Now it would need a boolean flag as well, which would tell it which would be the uh which would be the information that can be used jacket or not. So if, if there are certain features within the plug-in which are not supported by jgit and they are being used by the user, we would not recommend jk, even if it's uh optimal according to the repository size.

A

So I've raised uh two pr's for that, one in git, plugin and one and git client plugin mark, I think, you've seen the git client plugin you thank you for creating those tests um and uh for the git plugin one. I think uh I haven't created the tests, but I have created a place where I think I can show that part of code where we are.

A

Where we are using git tool choose and how we are going to just the flow of how we we are determining um every. How are we taking the decisions? Just a.

C

A

A

This is the comment which would so so in the git scm class. We have a cloud, we have a method called create client, so it is the place where we uh create the git client for any operation we want to provide so so what I'm doing here is here is the unsupported command which is being instantiated before any creation of client.

A

I am using a new decoration method. It is, it is only implemented by those extensions which which wish to convey the information that hey, I am not being used by. uh I cannot support jj, and uh so I am implementing this method, so so these extensions, what they would do is they would add that information to this instance, this object, instantiated, unsupported command, and once we have that what is going to happen is I create the estimator here.

A

I am getting the url uh and then here when I'm calling the get instantiating the get to chooser, it is going to take the url and also it is going to take a decision taken by the unsupported command, which is to determine the support for jet.

A

How it takes this decision is basically we we have a flag if we see that we have a non-null value for any of the feature which is not supported by jgit, we would just say cannot use, uh we would return a false for jk and the git tool then can make a decision can make the right decision of not recommending jegadif uh if any feature is not supported by uh jet, and this way we would not break any existing use cases.

A

So if a user is expecting that, for example, if a user has chosen git lfs pull and if we were not implementing any kind of functionality like this- and let's say the repository size is a small repository.

A

What would happen is that for him earlier gate, lfs pool was working, but now it would not because we've chosen jgit for him, but since uh now we're taking the decision additional decision of checking if the feature is compatible or not, we would not break as far as I've thought about it.

A

We would not break any existing use cases with the introduction of this uh decision so and then, ultimately, when we have uh the estimate the get tool chooser, we would just get the git tool and then we can feed it to the um to get this. This is the method which creates the get client and then we get the client. So this is how I am. I believe this is how we would we would use get tool chooser anywhere within the git plugin. These would be.

A

These steps would be would have to be taken to ensure that we do not break any existing use case and sideways. I provide the optimal implementation we wish to so um we have the prs for it. The code is written. I I of course I need to add the tests. I don't have the test now I we we would have uh say I I've seen that we have at least 10 features which are not supported by jgit.

A

So I guess I would need a lot of use test cases to cover all of these cases and how, in those scenarios, we are not recommending what we should not. uh So that's a task for me. Apart from that.

A

I, the second thing which which is important for the get 2 chooser, is the extensions, the extensions which will implement the extension point we have provided, which would give us the information of the repository size through rest apis so with the github uh so currently how I implement before the demo, I implemented an extension in the github brand source plugin.

A

So the way I implemented it was that I I sent the github repository url to the plugin, and then I asked the plugin to give me the size of the repository, and so I did not involve any credentials user credentials throughout this process. So this request rest api request was going unauthenticated as an unauthenticated user. So this was. This would currently work only for a public repository not for a private private repository.

A

That is the first uh I would say um a feature which is missing that we, we are not supporting private repository estimation. The second thing is that I was uh talking to um liam liam is the repository owner of uh get a broad source plugin. So he was saying that providing the github url is not enough, because there are enterprise servers which would have entirely different custom, customized names which would not even contain github in this way.

A

So we need to think about a different way to provide them a unique, um I would say, maybe combination of the repository url and user id or username.

C

Wait: wait, wait, wait! A second okay, I'm not understanding that one at all, because if I'm corporate, if I'm running github enterprise, it's got a url somewhere and the repository url absolutely encodes the full information. It could be.

C

My my server.example.com slash, marquee wade, slash some secret repository dot git, but but that url absolutely does identify that repository. Can you help me understand more? What liam was trying to tell us, because the https url to a repository is absolutely unambiguous, that that is a repository. Maybe inside a corporation it may not have the word github anywhere, but it is absolutely unambiguous.

A

So uh so, so what so I implement how I implemented that validation, that okay, is this a github url or not? I just I. I just put a check that if the string contains the word github.

C

Okay, great great so liam's right, then yeah, because the github enterprise installation at ca technologies. Where I worked, I don't think it had the name github anywhere in it. So yeah, okay got it.

A

So that is what he was saying that we I need to change the way.

C

I'm validating yeah that that's! Okay! That's fine! It's not that it's not that he was saying. There's more information needed! It's just that that the url may in fact not have the word github anywhere in it.

A

So what should we should we validate the url if we have the url, so in that logic of validating it, should I send a request to it and then check if it's.

C

I just pass it I'd, pass it to the let the branch source decide, I wouldn't put any extra validation on it. Personally, I would just let the branch source decide what it does with it, but.

D

You would know if it's the github branch source plugin or like the github branch source plugin itself would know that it's again a branch source right, so good.

C

D

Responsible to decide yeah, because the other tricky bit here is that you need an api token. Instead of like an ssh credential and some people will try and favor ssh credentials you may not have I don't remember if you can configure that without an api token, I don't think you can actually anyways.

C

And those branch source plug-ins know all about those complexities right and they worry about those complexities. Oh, I need to make an arrest request, but all I have is an ssh token. How do I do that? The answer? Is you? Don't you have to get another token, so yeah go ahead? Rishab, sorry! So forgive the forgive the side trip. I was just concerned that there was something I hadn't understood. Thank you.

A

I uh I think uh so I would need to um because I would need to implement the extensions in any of the brand source plugins. So I think I would look more into how I can do it look more into the code and implement them uh okay. So so currently, what I was thinking was to uh first implem for consolidate the implementation with github and merge it. If it's available with them, then in gitlab, then we we can maybe proceed to other plugins as well.

A

Now uh I had one question with the with the the tool chooser which mark you commented about it about a thing that we can have the cash if the. uh If we have a multi-branch project and it it has a cash gate. Repository directory uh a pipeline, a project which is implement a separate project which is implementing um a checkout step, can also access that cache repository.

A

Okay and uh how would that happen.

C

I I just the the master, the jenkins, the jenkins master, the jenkins, probably eventually called controller. The jenkins controller has a caches directory and it doesn't care what context you're asking for asking for information about that cache directory.

C

So if the url, the repository url, is known to the jenkins cache, it can be, its size can be answered whether or not the job asking the question is multi-branch or freestyle or or multi-config or simple pipeline.

A

Okay, so, as far as I know, we need this functionality of locking a cache and then examining it is provided within the scm source uh class. It's not provided within the gate, sem class.

C

So right because I think implementation, I think the cache implementation is actually in the git plugin.

C

If you, if you look at it.

A

Okay, okay I'll uh I haven't you know, I think I haven't looked at that in detail. I would talk more about that.

C

Well, I I think you're actually already using it. This we we'd had a conversation in one of your poll requests about making sure that we did not bother to acquire a lock on that cache, because all we're doing is asking about the size of the thing, and you had. You had implemented in your pull request that it was opening the cache but not require not acquiring a lock on it.

A

But but I'm using it when I have a git sem source object: okay, I'm not uh that's how I'm using it.

C

I'm sorry, I'm not I'm not following it. That still seems like a perfectly safe thing to do. What did I miss help me understand so.

A

What I'm saying is that, if I am, if I'm trying to instantiate get to choose within the git sem class, would I have to get sem source object there available?

A

No, because that's a different project right as far as I understand gear, sem source is a class which is used when we are trying to scan multiple branches. It's a separate multibranch project and when I'm trying to use uh let's say a separate pipeline project where I'm just checking out the branch, a single branch I am using the sem class and uh what I'm trying to say is that would I have the access to that object?

A

How would I access the cache, then.

C

I I think, just by asking for asking to to see if there is a cache, if a cache exists for this url for this repository url and if it does read it.

A

Okay, so uh I let me just open the get to chooser for what I want to say: I'll, communicate better with it.

A

How, if I want to access the.

A

Okay, so uh mark, if you can see here when I uh so I have a function which is which uses the cache and determines if the size of the repository using the cache. So I expect a qrcm source object and I use that object to get the cache entry and then I use a static method provided by abstractgm source to get the cache directory. So what you're saying is that I can oh, I can use this to.

A

I can use this in git scm class to get the cash uh directory. I.

C

Think so isn't isn't the key there, the cash entry key. Actually the repository url.

A

uh I don't I'm not sure.

C

A

C

My hope was that that that cache entry value is the repository url. If I'm wrong on that, then we've got to go. Look what the what the index is into the cache, but I would be I was expecting it was either the repository url or some variant of the repository url and.

B

C

You can just ask the thing through that static method get cacheder. Do you have this? If you don't.

A

C

Estimate with it, if you do I'll estimate with it.

A

I I understand a point now. I was not aware that the cash entry is actually the repository url. Okay, I just implemented it the same. Okay, I'll I'll uh I'll, first see what the catch entry is and then, if it's the repository you are, then we don't need a source object to to get the cache.

A

A

So I think this is it for the get to choose it, um the things we have to do so now. uh The question is that what so planning for phase three? What uh what should we do? Apart from get to chooser and the release we have to do for the fix? We did redundant fetch?

A

Well, the first thing I thought was to implement git clone, maybe, but then the reason to implement it. One of the reasons to implement it would be that it's it has a better performance than get fetch and I've shared a document indicator channel. Where I've come I've. I wrote a lot of benchmarks. I wrote three benchmarks, different different benchmarks for uh to compare their performances. I only have uh the result for one here, but for all of the three benchmarks, so the first benchmark we will see.

A

This is a test for a single repository uh which is around 300 mb, and what we see here across different platforms is almost similar uh performance, there's a second difference, but there's an error rate of one second as well. So cannot be too sure. Even if this is a one second difference here, then I ran another benchmark with where I varied the size of the repositories to see with different sizes.

A

How would git clone versus git fetch work, and I did not see any difference in that case, I also create I experimented with running the same benchmark with uh on the centos seven with git one pointed, but there is well. I could not see any performance difference noticeable performance difference. It was around uh half a second, the difference between git clone and git fetch.

A

Now this can mean two things: the first that uh there's no difference the second that my benchmarks might be wrong and why I would say so, is that the reason why I'm, how I'm implementing git clone so I'm using the launcher, which is provided, which is the implementation for uh cl? I get uh to perform uh to launch a git operation, so I'm using it to call git clone. I hope that I'm that's. That's! That's the way to do it.

A

Okay, so if that's the way to do it, then I'm then the results say that there's not much of a difference. So then, would we want to implement git clone? Is there a use? Is there a benefit uh to implement that feature that operation instead of getting it plus get fetch?

A

C

For for me, there is for me there is not given that give. I would much rather invest your time providing git tool, providing the the branch source implementations to giddy, to github, to git lab to bitbucket to to tulip to to to the other branch source providers. I think it's much more valuable than spending time on this on adding git clone as an alternate implementation of a subset of the capabilities that the plug-in already offers.

C

The the git clone can't do all the things that init plus fetch can do, and and therefore, when we, if we were to implement clone, it would be purely a subset and a subset that the user then has to be very intelligent and decide. Do I want to use init plus fetch, or do I use clone, and if I use clone, do I get the benefit? You know it's an awful lot of analysis that I don't think we really want the users to think about.

C

Okay, understood, okay, I'm open to others disputing! That's that I don't I'm, I'm certainly not the sole mentor here, but I my sense was: this is not enough of a difference to justify your investment in it.

A

Okay, so um the other mentors have anything to say and definitely active markets for user. It will be too difficult to make that analysis and go with any particular decision.

A

C

I mean I I would. I would beg you to include this in a in the summary report, because I will refer to it frequently when people complain to me that they want clone implemented. It's like. No, I'm sorry, we really did a rigorous benchmark and the benchmark says this. It's not just me saying it. The benchmark says this. There is not enough difference to justify our energy to to write a whole new implementation.

A

Okay, okay, mike I'll I'll, even add the other expert benchmarks. I did I'll add all of those results to uh have a more comprehensive reporter, then I'll, yeah and add that so uh so, if you're not doing that, so what my question is that do we want to implement and consolidate what we have here and then that is what we deliver for this project. Or do we want to look for something else as well for this phase? Would we so would we want to do we want to explore another area of performance improvement?

A

uh I frankly, I'm not sure where I could look for that as much as I've seen the git plug-in and the benchmarks I've done. I don't know where I can find. Where is where is it? Is there any area where I could find a sizeable difference between performance or something where we could work to improve it?

A

One thing I had in my mind was uh was: was this pr I've seen, which I'm not sure if we would want to look into it? Where did it I lost it? I think I think the number is 500. You can jump 502. I think I've written it- okay, yes, so uh this is something uh the mentors can decide if, if we would want to work for this idea, using the git sem file system, cache as a reference protocol and flow.

C

You are, you are a brave brave man that is very impressive. This was an entire gsoc gsoc project proposal to consider taking this thing on so, yes, anything you can do to help this move forward would be great, but for me, there's much more value in first assuring that we get the branch source plug-ins all the way to done so that they're contributing an answer to the to the question to the the tool.

C

Chooser question right for me, those- and I I expect that's already going to be challenging for you, because the giddy, the giddy plug-in the bitbucket plug-in the gitlab plug-in the github branch source plug-in each of those will have its own different and unique things that you're going to discover and- and I don't even know what the tulip or tulip plug-in does- but it is also a branch source.

A

Okay, okay, so um so so then what we can um so the deliverable is for us, then uh the get to chooser with the extension with the support we are promising with the plugins we have. I think that should be the first priority for us, because we want to uh when we release that feature. It should actually work, as we have promised it to then.

A

If so, if we are able to do that within a certain time period- and we still have some number of days where we could work on something else, would we think about this, or should we focus on first focus on prioritizing the release of git tool, chooser and the future? The extensions, the support and everything.

C

Okay, so so my preference- uh maybe I should be quiet fran, you want to give your preference first. I others others are welcome to chime in here and that's I. I tend to be opinionated and loud and I don't want to overwhelm others with my opinionated loudness.

B

No, but I think I think that you're right um before it's been reopening this box, I would just try to yes, I would try. I would try to uh to finish all the all the effort with the with the brown sources back, because I totally agree with mark.

D

Yeah, I think I think that's gonna, make sure that you kind of can keep focused and finish that off and then you know if there is extra time, then maybe we talk about like what else. Okay.

A

Okay, I understand and I'll uh okay, that's how we'll move forward. So uh I think we have finished the time. I just have one question and that's question that that question is uh related to jay gate so mark I wanted to ask: when does a user? You do people even use jet, my so the reason I'm asking the motivation behind is that we are providing a feature which uh this git tool chooser, which would uh would work well if a user is using jget for a large size.

A

Repository then, is the that is the scenario where any user would see a noticeable difference in performance after implementing this. After adding this feature to the plugin.

A

So do we even have those cases where people are using jget for, let's not just say the size of the repository? Maybe that might not be something we can know, but why do people use jget one one. One way I know is one reason is that for uh places where gate is not installed, jk can be used, but then I don't understand. Why can we not in? Why can we not have get in any in in any environment.

C

A

Why would we use jet.

C

So so I'll give you a very specific example: ci.jenkins.io builds the jenkins website as one of its tasks. It has infraslash jenkins io and in that things, output, it's cloning, a 60 megabyte repository using jgid and that 60 megabyte repository keeps growing. By now it may actually be more like 70 or 80 megabytes, because people keep adding pictures to it.

D

C

And so, as as we may reach the break point, the point where the tipping point where command line git would be much better for that repository than jet. uh We may have already reached that point. uh The reason that the original implementer of that thing did not install um command line get is he, I think, was fundamentally lazy and didn't want to right. He thought I don't want one more tool on things: let's not install command line git and the choice to not install command line.

C

Git turned out that it broke all sorts of other testing, because there are many different components which assume command line. Git is available and therefore we have command line git. We also are still using jgid.

C

So so so I have an example. Now we can, we can call it contrived, because I don't we don't have hard data for how many installations have enabled jgit. At least I don't I'm not aware of a way to get that kind of hard data, but but it's a valid point that if they don't have jget enabled we probably can't reference, can't suggest j get to them or if they don't have cli get installed. We can't choose cli, get.

A

Yes, I was, I was just wondering uh how it would affect how this feature would affect uh the users when they're releasing this and and how much of a practical usage this feature would have. But, okay, I I think I that's okay, so, um okay, so I I'm going to work on um the test. The first would be now since I think the pr for the git tool chooser is complete now in terms of the features it wants to uh provide and the decision it needs to take. That is complete.

A

Now it was missing the j get compatibility uh issue, and now that is also I I've raised a pr for that. So now I can move on to uh consolidating the support for extensions. I have already raised a request for github brand source. I would um first I would try to merge, get that merged, and then I would- or I would probably work on git lab and take another plug-in as well so I'll.

A

Try to that is my first task and the second would be to add test cases to more test cases to uh test pull request and uh the second one I've raised the unsupport command feature so.

C

Okay, and so so is your sense that unsupported command has shown well enough for you, it's working the way you expected. We could safely merge it to get client plug-in so that you could have a a readily usable build that has get client plug. It has has the capability in it.

A

So I think locally, it doesn't doesn't matter to me because I would have uh the snapshot jar in my m2, so so I would say for the unsupport command and supported command, I I think it doesn't. It doesn't have anything which would require intensive interactive testing in the get client plugin.

A

But again it's it's, uh it's a new command. Is it? Should it be a command? Should it be? Something else should it be shifted to the get plugin not be kept within the git client plugin?

A

My point is that if you people could review it and and uh I'm I'm, what I'm confident about is that the feature is providing what I need it's providing the information I need, but um other than that, is it consistent with how we develop other classes in gate plug-in, and should this this be the way it should be developed, or uh so I just want the mentors to review it.

A

That request and um I for my personal development I would have the snapshot. uh I actually.

C

A

Yes, yes, ma'am.

C

You you answered it perfectly: it's not blocking you to not have this pr merged, that's that is great okay, so that that gives us the flexibility to review it to think deeply about it to compare it in context, perfect! That's all I needed.

A

Okay, yes, so I can, I would have uh a snapshot jar for the client and then even if I have I implement the extensions on the brand source plugins, I would have their uh snapshot deposit uh charge in my m2, so I don't think I would be blocked by any pull requests um status.

A

So yes, so I think I think that is it then, for this meeting.

A

Anything else anyone wants to discuss.

C

Are you, okay, with the things that have have come come out in this meeting? I I know saying that we're not doing get cloned, probably is a hard one, because hey that was that's a major chunk of implementation. Are you okay, with with that approach,.

A

Yes ma, I think for me as well. If, for me, the the only reason to uh uh to recommend new features is that I need I get to code more and I get to learn more, but then what I have. What I think I should remember is that we have to think about usability.

A

uh That's the most important thing. If, if I I just think about writing more code and not think about, if that thing is even will be used by the users or will be used in any uh place in the git plugin, then I don't think it makes sense.

A

So um so I think we've had a logical discussion about what we can do and and what's the first thing we should do, because if we are promising a new, uh a new feature which improve perform in certain cases, at least it should do that first and uh then, if we have time, then um I can look into we can discuss, and if we have the time we can look into more challenges. We could solve yes, so my only issue was that have I done enough for performance improvement within the get plugin?

A

So that's that's what I was trying to explore and so uh and I think that was why I'm asking these questions. Yes,.

C

Great so, let's proceed forward then um and yeah. I I think I think we've got lots and lots of work yet remaining I'm a little worried that we may not get to release by end of end of the time period. Even so so there's there are so many things so many parts and pieces that just fire release we we we will certainly need to release, get client we'll need to reskip plug-in. We probably need to release github branch source and bitbucket branch source and giddy and git lab, and yes,.

A

I understand the concern mark and- and I think we should move forward in full stream- uh developing them first and consolidating their releases. Yes, that's that's! That's a great plan.

C

All right, thank you, I'll, post, the I'll post, the recording thanks.