Jenkins Google Summer of Code Office Hours, 4 Aug 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: GSoC 2021 Git credentials binding 2021 08 04

Description

Jenkins git credentials binding office hours August 4, 2021. Topics included discussing issues with private keys, using the sshd-core library to resolve the primary private key algorithms, and dependeng on a plugin vs depending directly on sshd-core.

Meeting notes are available at https://docs.google.com/document/d/1gZneYIDWrT5S-1ACG641wfvxs7vnDC0RCYqy-EuuhwY/edit#heading=h.m0lph2u36pvh

A

Cool so recording and let me share my screen.

A

A

How's everyone doing.

A

This is the credentials binding project for august 4th 7 30 am india standard time. Sorry, I asked a question and then I forgot I didn't introduce the meeting for the.

B

Youtube, I know I've been great, although I have to admit I did not complete the things that I had envisioned doing in terms of preparing to test. So my weekend was spent on other jenkins things not on not on our project.

A

C

I uh tried reading passwords, protected, rsv keys opens by generating them from open, ssh and then uh using the ssjs client they've provided to connect to a server. But I was unable to do so. I'm not sure why it was not connecting. It was giving me a timeout exception, but it was not working for me.

C

Yeah, the library is able to read the keys faster fastest, directed or not fast was directed, but I was not able to connect and like establish the ssh connection.

C

C

uh I believe harshit has found another library.

C

To do the work.

A

Sure would you like to talk about the library our.

C

I should get on me if you're.

C

C

C

B

Enough, I don't have the thing that would allow me to ask him to unmute.

B

I thought in the past I I'd had an option that would allow me to invite someone to unmute their their, but I don't see it here, so maybe I'm just remembering badly or maybe it was only from a webinar.

A

Oh, I got it. Oh, it looks like you joined us. Another account.

D

A

Maybe audio problems.

D

I'm connection.

C

You found another library to do the work right.

D

C

Yeah I was unsuccessful to create, like I, was able to read the recipe but not able to establish an assistant's connection. Then this is using.

D

Yeah, so I found out that the the fingerprint of the private key that is generated using the decrypted privacy that is generated using the ssj library is different from the fingerprint of the private key. That is, you know encrypted. So I think, due to that, we are not able to connect, so the keys are different, so the server won't be able to recognize a match for that private key with the public key.

C

But um I just wanted to ah before you introduce the new library. Did you ever try to read the keys and use the client called ssh client provided by assistant to connect to a remote server using um you know any of your ssh keeper.

D

No, you are saying to that. I should connect to the ss server that is good to use using the ssh library rather than the command link.

C

Yeah I mean the I thought that the fastest way to know if ssj will be able to decrypt the keys correctly, and then uh you know they it is going to provide this keys, which are going to work, is to use their own client to establish a connection with any server any you know any place which has a 22 port open and we can establish a connection with them. If that is possible, then those keys, we can be sure that ssj is not creating an issue for us.

C

I mean, um but you found a new library right and yeah yeah and you were yeah. Please move forward with me.

D

E

New library is.

D

You know the apache shd, so even jenkins has a plug-in for that. It's called ssh server. So currently, I'm facing a dependency issue in the git plugin do do this, because the assistant server is an implied dependency and this. I don't think that there is any direct you know to define.

D

You know explicitly defined dependency or ssj of the sssd so and it's using a version that is below.

D

3.10 so, and the code is very much different from the one. That is that we want to.

C

Use have you used? Have you used this library to uh use rsa passwords connected keys and then were you able to do that.

D

Yeah, it worked with them and I was able to establish the connection as well.

D

So but the issue I'm facing is you know the dependency issue, so I think I mentioned it on the gita chart. So so what should we do next right? Should we define this dependency explicitly in git plugin, and if we do that, then we have to you know, increase the jenkins version great more than two point above I think it's two point. Eight.

D

But currently the conversion git plugin is 2.2, something like that or 2.6 like, and I'm not sure about that.

B

Parshat, are you saying that that a a newer version of jenkins core has the version of sshd that you need.

D

B

Oh good, okay, so so that's a great excuse for us to and what what version of jenkins core has that.

D

It's two point: let me check on the bitter chat I mentioned that.

C

D

Yeah 2.289.1 or 2.282.

B

Okay, so so, if we upgrade upgrade our dependency to require jenkins 2.289.1, then we get the sshd that you need.

D

A

Yeah- and I think I think, you're saying that either one of these ssh server plug-ins would work.

D

Yeah, so these have the version of the apache sshd library that we need and that supposedly decryption of the open, ssh piece.

B

Yeah so then yeah it seems like a simple answer to me. We upgrade the minimum jenkins version to 2.289.1.

B

A

B

Mean you have a preference.

A

Mark on whether they pick we pick uh so it looks like there's two plugins, there's 310 and 304.

A

um One of them is a higher jenkins version, and one of them is a slightly lower one, the other one's 2.282, not sure. If you have any.

B

I I've biased in the past towards choosing an lts version, and in this case I don't think, we've got a lot of users that would be on 283 through 289, but not on lds right. So for me the preference is lts.

B

Yeah that makes sense because weekly I mean weekly, you should upgrade weekly actually because, for instance, 2.300 has a security fix in it and if you're, if you're, choosing to run weekly but you're still on 2.282 you're in danger.

B

So but if you're on 2.289.2 you're not in danger, because the security fix is backported to it so making it 2.289.1 seems very reasonable to me, and I knew we knew we were going to have to eventually upgrade yeah. There will be people who say, but I want to run the credentials binding without having to upgrade to 289 289.1. And the answer is sorry. You can't.

B

So so how should it feel it sounds like, then? What your technique would be is you'll set the jenkins dot version in the uh in the palm to be 2.289.1, and then you can use that that library that we that's available.

B

I mean certainly there will be people who say hey, I cannot upgrade to 2.289.1 and if, if we had, for instance, to do a security fix, we may have to deliver multiple versions of that security fix so for people who can't go all the way to 2.289.x.

B

But I'm fine with that. We've done that before.

B

Does the git plugin already rely on ssh server plugin? I don't, I didn't think it did directly, but um let's check I mean that may be a circular dependency. I'm not sure what that will mean, because I would expect the ssh server plug-in probably relies on the git plugin.

B

A

At just looked at git plugin, it's not in there.

A

And I look to get client plug-in and it's not in there. Oh I'm not sharing that screen.

A

B

Okay, get get server does not depend on the git plugin at least not directly, declaring it. Oh that's fun and the git server plug-in requires jenkins 2.289.1 as well, so we're in good company.

A

Harshan, I guess the one question I have is: do we want to take a dependency on the uh ssh.

B

Server plugin, so so, when I look at I'm not sure it's ssh server plug-in, can you could you take a dependency on org.jenkins.ci.modules.

B

Sshd and use that dependency, so I think now that's the sshd plug-in that you're looking at okay.

E

B

Okay, what's what's the what's the? um uh What do you call the address of that thing? The coordinates of that thing you're looking at. Is it org.jenkins.ci.modules yeah? It is okay. So that's the thing that that ssh server or that git server plug-in depends on, and I think is so harsh it is- is sshd from apache or from this code. The thing that provides the method you need or do you need something that's actually in the git server plugin.

D

The bit server plugin actually has the yeah. So if you see that the online 53 it actually has the latest version of the library that we need so either we can depend on it directly or we could depend on the.

B

Oh okay, so I'll all you need is org apache sshd.

D

B

Oh good, then we just should depend directly on that and if that means in order to but in order to make that load, then we have to increase the the jenkins version number to 2.289.1.

E

B

C

uh Why do we need to increase the jenkins number two? This is. This is an external dependency right in an external library.

C

E

We are choosing to use ssh.

C

D

Now we have to increase uh if you can see on line 34, but also, if we, you know, define external dependency that is sssb apache, then in the mavin dependency tree it will be used rather than the uh it will be used in for the sshd server or in a plug-in jenkins plug-in.

D

So, instead of because currently that plugin is using the library version 1.7 so after we externally define it, it will be updated to 2.7. So in you know, implicitly, we are changing the library dependency of that plugin. Only.

C

No, but uh you can exclude the depend since you're going to directly use this library this dependency. Then you can exclude that from the the plugin, which requires a newer jenkins version, and then you can use directly this time. I don't understand. Why would we need a new gentleman's version for that.

C

If you're, excluding it from that plugin and using this instant.

D

Plugin or the library that the plugin has or yeah.

C

Ssd code, sshd code: this is the exact library. This has the code, the api, the functionality. You need to decrypt the key right to read the keys.

D

B

So so back to richard's question, but to ask it in the form of the question: is org.apache.sshd version 2.7.0 enough for you to have the api you need harshad, or do you need something that is? It is yeah okay. So then I I'm prone to agree with with rashab that we should at least try to just depend on that library. If that still requires us to upgrade jenkins.version.

B

That's fine! It's acceptable to do that absolutely, but I would rather directly depend on that apache library than depend on a plug-in that includes that apache library.

A

Yeah, because what will happen is, if you depend on the plug-in directly, when someone wants to install ssh or sorry when they they will need to, they will basically get sshd plugin installed too, along with the git plugin.

B

Well- and I think sshd plugin is an unavoidable plugin because it was bundled in core. Oh was it oh, I think so I think there's I think, sshd plug-in is one that was discussed just recently on the mailing list with wow. Why is this there? I don't need sshd, and the answer is well because it was bundled in core and years ago, and thus it's still its api still has to be available in core okay. Well,.

A

D

A

See this active.

D

Poem of the git plugin you could you can find that plugin, yes, the plugin is defined, but the version is changed.

B

So you're saying that the apache sshd core library is already in the the get plug-ins dependency.

D

Tree ss ssd plug-in jenkins.

D

D

I was talking about the jenkins ssd plugin, no on the actual apache library.

B

Okay, so so I I apologize, I must have missed a word, so the jenkins sshd plug-in already includes this, and I think that's what we're looking at right now right, yeah, that's what what justin's showing us is that that apache library is included inside the sshd plugin agreed, but my thought was: if all you need is that apache library, then all we should depend on is that apache library, with its its specific version, number.

D

Yeah, the kit plugin, you know implicitly depends on that. Sshd plugin. So.

B

D

It does I was saying that in the effective poem of the git plugin, you will see the ss jenkins ssd plugin dependency available there. All.

B

D

You know low, lower.

C

But we can override that right. We can say that we don't want that. We don't want ssd code from that plugin. Instead, we'll use our room.

B

Yeah but but I think I think I'll go ahead.

D

That would mean that we are upgrading the version of the core only like we are not replacing. We are upgrading the version of core, for this jenkins is deep. Plugging.

C

It's not just ssd code right, it's it's! It's a collection of multiple libraries and.

E

C

Choose to use a new version of another library. How would it increase the version of the ssd plugin.

D

Because in the dependency tree, if we look at the library that you know the dependency that is on the top will be used rather than dependency on the bottom, so the plugin that is using a dependency of lower version, will be upgraded to higher version.

D

So explicit, uh so we can implicitly we are, you know using a lab core ssd core of version 2.7 0 only instead of the lower version that is 1.7.0.

B

Opening ah look at that.

B

Okay, I think I understand what you're telling us arshad okay, so I did what you s, what you described, which is, I ran maven dependency, colon tree and in the output of that there is a dependency on org.jenkins.ci dot modules, sshd version 2.7, which then itself has a dependency on s on the apache library, core jar, our core sshd cord 1.7.0.

B

Now, if we upgrade, I'm not sure if this actually works from maven, if I force it to jenkins 2.289.1.

B

Does it then already tell me, hey, we have a newer version of ssh decor, just a.

B

B

Odd, when I tell it to depend on to to require jenkins 2.289.1, it doesn't even have a dependency on ssh decor, interesting okay, but you you've done the experiment harshit, and you said that by upgrading the by forcing the jenkins jenkins.version to 2.289.1, you get the you get access to the sshd core library version. You.

B

D

Dependency explicitly and as a as registered also pointed out, I was defining the apache ssd dependency explicitly. Both I mean I was trying to experiment that so that will indirectly tell us to upgrade the jenkins version.

B

Yeah- and I think I think that's the right approach, I think we should accept that in order to get this functionality they have to at least be running jenkins 2.289.1, because we need the version of sshd core that apache provides, and that is that is available only beginning with 2.289.1. That's that's.

B

C

So uh how should with this library is? Was there a specific algorithm.

C

This library solves the problems we face with ssj.

D

Yeah, so if we are using this library, so the working keys are actually for in openness, j format, they will be. You know, rsa e c d, s, a e d, two, five, five one, nine and yes, so, actually all the algorithms encryption, algorithms that are supported by the ssh keygen utility.

D

That's and they were working also.

E

That's great: we don't even have to use another library to do that.

A

Yeah, that's great.

D

Yeah, I was thinking of you know, using the there's, a pem format, uh support for pam, formatted keys or pkcs8 format, keys in the lab in that library as well, but.

D

The there's an issue opened on the you know mailing list that says that the pkcs8 encoded key is not yet supported. So, as I have to switch back to bouncy castle plugin.

C

Okay, friends, you're talking about pkcs8, encrypted keys.

D

I'm encrypted yeah, I'm encountered keys, yes, but they are not supported in the library, sshd library, so.

C

But um so my question is that do we want to first focus on the algorithms or on the use cases which, which will be 90 of what we're going to serve, or I'm not sure how? How many? In how many cases we're going to see the pkcs8 encrypted m, encoded format, keys.

D

I was just doing an experiment with that, because yeah this is ssd. Library has a lot of functionality and there is a lot of support also. So I was trying to figure out if I could mostly use all of that and instead of depending on the bouncy castle api, but no, it was not supported in that.

A

Yeah I mean I think, like like we talked about the last time. You probably could start with. You know just loading in support for the top ones um and if we find that you know pem supports not working, we can document that as unsupported and that's a jira ticket or something like that for later right.

D

Okay, so I mean, should I then use the sshd plugin or well? Actually we use both. Then we have all the keys working on all formats.

C

When you say both, you mean ssd and bouncy.

D

Bass, the library.

C

There's no additional cost of using bouncy acid. Then we should I mean if everything is working, then why not I'm not sure what my ingestion.

A

Yeah, I guess from a from a process and work perspective like if it's easy for you to get them all in one pr, like maybe that's okay, I guess one thing I could see is like if it's hard to start getting the bouncy castle stuff, then maybe it's a good idea to start with the the most critical algorithms to cover and then do those in a pr. uh But then I I also realized, there's some testing and stuff like that involved too. So those are the things that I would think about for that.

A

If that makes sense, yes,.

D

So uh one doubt I have is that, should I you know explicitly define the this: is the plugin that the jenkins module, ssd plugin right in the poem of the plugin.

B

Sorry I asked that question again. It was, should you define explicitly define the version of sshd dash core in the pom file for the git plugin.

D

On the asset, then consistency plugin should explicitly make a dependency indicate plugin, or should I use the apache.

B

D

B

Me I would use the apache sshd core to to make the the dependency as narrow as possible, so as narrow only as only as wide as you need. Yeah.

A

And, and also explicit on what you exactly need to it's kind of a nice.

B

Practice all right now now no guarantee that that still that that will work because of the nature of that sshd plug-in being a formerly bundled plug-in, you may still be forced to to set the jenkins minimum version to 2.289.1.

D

Yeah, I will have to.

B

Okay, good and and just to reiterate, that's that's perfectly. Okay,.

B

We want people to upgrade, and this is a good motivation for them to.

B

A

Cool, so I'm going to put that in our notes here too switch to minimum jenkins 2289.1 and depend directly honest, hd, core 1.7.0 and you're, going to do that in the.

B

Git plugin right: no, it will depend directly on sshd core 2.7.0. I think it is. The current version is 1.7.0 and that's in two yeah 2.7.

B

I bet uh and then that'd be in the git plugin right, not in the git client yeah.

A

B

Sshd justin, just that extra h, you got me thank.

A

B

A

We can do it.

C

Hey, I have a question um I I was also looking at the dependency graph and uh I can see that, as as have mentioned, that um in the jenkins pool jar, we're getting sshd code, sshd plugin as a test score, can we not remove this all together? Do we need the ssh plugin for anything.

B

Do we need it? Yes, in order to run tests, is that what you're saying is that it's a get plug-in test dependency.

C

I was uh yeah okay, we use this depend. I just thought that this is being bundled, but we don't use it.

B

No I'm well so the j, the git plug-in, definitely does not bundle the sshd plug-in. So so I'm not sure. Maybe you need to explain to me more more clearly what you're, what you're meaning by the by bundling you mean the jenkins core, bundling it or the get plug-in bundling it.

C

Jenkins, we're getting this from jenkins code right.

B

C

And uh is the kit plug-in using uh this library.

B

As far as I know, it is not today.

C

Yes, my question was that: can we not just remove this uh exclude this and um sshd4 directly, the plugin all together.

B

I I I we may be able to, but I don't know what that will do on a jenkins 2.263.1 when the git plug-in wants to load sshd, core 2.7.0, but jenkins has already loaded 1.7.0 of the sshd core plug-in because of the built-in sshd plug-in. That's using sh sshd core 1.7.0.

B

Okay, so I would expect jenkins will say I've already loaded, sshd core. I refuse to load another one and then we'll be stuck because git plugin, though, won't be able to find the apis. It needs that are only available with newer than 1.7.0 harshit. I think I was describing it correctly. Do you need to correct something? I said there and tell me no mark. You made a mistake.

D

Mark you are right, there.

D

Myself so I know the error will be, you know no class, no class found exception. You know that was coming because the library that was loaded was one ssd, co 1.7 and I need the libraries of ssd core 2.7. So there was, I was stuck for. You know for one day whole one day depending on what why? Why is it this happening?

D

So then I figured out that this for dependency issues.

B

Great thank you and welcome. Welcome to maven's dependency tree.

A

A

So that sounds really positive.

A

Does that uh kind of help you, with all the questions you have for for that specific one harshet.

D

Yeah you know so, for I mean I'm currently focused on this issue only so I have not even worked on the issue, but I'm just focusing on this issue.

B

Sure but but this issue is, I think this is a major victory that you've reached. As far as you have keep going, that's great, and did you see harshit that we have one more person who's interested in in the work and did some experiments? We can ignore their results if we need to, but I was delighted it's great, that we got people who want to use your code.

D

I mean I missed it.

B

So I I had closed the bug report from someone who asked for the ability to add a special case to the jenkins git publisher. That would allow it to push push multiple tags and the the person actually submitted a pull request to the get plug-in saying: hey, here's, the implementation and it was a good implementation. They did a good job with it.

B

But I closed that pull request and said: nope we're going to do this with with this google summer of code project, and so the person then went and took the latest build from the pull request and tried it.

B

So you'll find a comment on the pull request where the person says hey. It didn't work for me. Here's this, but given what you're working on with sshd, I'm not at all surprised that it didn't work for the for the for this person.

D

Yeah, I have to see that comment and the progress that he has made.

A

B

One? Yes, yes, that's it! That's the.

B

B

Yeah and- and I pointed the user and said- hey use, username password, it works now and he said no. I have to have private key and this is his effort trying to work with private keys. So that's great nice yeah. Yes, it didn't work. That's that's not a problem. The fact that we've got one more person trying it is really great.

A

Mark did you have any opinions on um the pem format and including bouncy castle in the in the next pr, or do you think that should be like a separate pr? What's your.

B

Yeah I'd rather I've, never once in all the time I've dealt with private keys, never once used a pem format, private key with with ssh. So I don't. I think that can be separate. Let's get the other things working and be confident we're tested and we've checked them in all sorts of configurations.

B

So, at least for me, I'm not worried about pem format. Right now,.

A

That sound okay, harshet.

D

A

You get to deliver more value faster.

D

And I will try to make the pr I mean I will try to make the pr by today for the open, ssh format, support.

D

Let's hope for the best.

D

C

Are we also? um Are we also covering the cases or um thinking about the cases when ssh key gen did not produce open, ssh keys before this format was adopted?.

B

uh I think I think we'll get that as a result, we'll get test cases for that, based on the the the tests that I intend to run with generating keys on centos 7 and on debian 9, because those are relatively older, ssh implementations.

B

So I think we'll get test data and even and what I hope the test data will then tell us is yes, it works with the sshd core. No, it does not, and if it does not, we, we will probably just initially say known to not work. You must use a modern key format.

B

Does that does that seem? Okay to you, rashav.

C

Yeah, yes, I think. uh Actually, I think I I'd rather look at what hershey's code is. I look at his implementation. um How it depends on these algorithms how it functions with these algorithms. Is it do you have to write different code for each of them like? Is there a different term?

C

I I just look at the code and I think uh mark that should be fine for us that we run those tests and then we figure out.

E

C

A

Any other doubts or topics we should talk about.

D

So the current here that I have opened for the asset binding. I should change that for open specific to openness.

B

Format, so our I'm not sure I understood the all the all the details with the audio, so were you asking? Should you alter the current pull request to only support open ssh formats.

D

Yeah and then for the bouncy castle.

B

For me, I think that that seems reasonable.

B

If that's, if that's easy enough for you, I that would be the way I would go if, if you find it easier to combine them into a single pr and still allow us to do, allow me and others to do testing that that's fine too, but for me, a separate pr with the bouncy castle. Work feels better just because I want to be sure that rsa and ed25519 keys work, because those are the keys that are important to me.

D

Yeah, actually, if I remove the bouncy castle, there are like four five lines of code that will be affected. I mean the rest of the code, won't even affect it.

B

D

Minimum there are four four lines of code: adding.

B

Well then, then, it may not be worth your spending the time to remove it that that's entirely up to you.

B

I I just don't want to be blocked, testing, rsa or ed25519 because of a format that I don't use now. Other people may have a very different opinion and say: hey. I use pem format all the time, but I truly do not recall ever using a no. Maybe I guess I did use one once because some of the cloud vendors give you a pim format key.

B

So I I guess maybe I have used one, but it's it's very rare for.

A

A

Do you have a preference on which way to go like whether to include it or not or.

D

I think I I want to go with the included one so.

B

A

B

Yeah and and now that I look at my config file, I was just wrong when I connect to the system 390 mainframe, that ibm has lent to us lent to the jenkins project, I'm using a pem format, key.

B

And it works great.

A

So yeah, I think if we include it, we can always just declare that you know something's not supported if you know, even if it goes in with the pr.

A

D

A

Sounds like we've got a path forward.

A

C

Else we want to talk about.

A

For any other doubts.

D

Oh man, I'm good for today, good.

B

Good shot mark, I'm good uh great for me as well thanks harshad, thanks very much and congrats on finding the solution to the ssh decor challenge. Yeah.

D

A

This is awesome.

D

A

Great we'll close her down early then.

A

Thanks everybody thanks all have a good one.

A