Jenkins Infrastructure Project, 27 Oct 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: 2020 10 27 Jenkins Infra Meeting

Description

Jenkins infrastructure team meeting October 27, 2020 with topics including helm chart deprecation, DockerHub terms of service change, Docker images for Windows, JFrog artifactory next steps, Jira upgrade plan progress report, and Azure cloud hosting topics

A

So yeah, can you start the recording mark.

A

Hi everybody welcome for this new jenkins infrastructure meeting um for today's agenda. uh We already put some um points under the duck and so yeah the first, the first one that we are going that I would like to highlight is um we still have a bunch of hem charts v2 that we have to upgrade um the biggest and the most annoying one.

A

Is that we're still using uh nginx ingress and in order to use the new um sharp location, apparently, they suggest to add, either delete the existing hem chart and to reinstall it um or to deploy a second one next to the first one, so we don't have any downtime for the application.

A

um We are exploring the second option with slaadin, so I had them. uh I had a call with him last week, so he opened a pr and basically, what we did is we tried to test his pr together and see how he could improve it. um So it's still working progress um but yeah. I expect to finish this either this week or next week, and then so I will be able to announce um a maintenance window, but the target is to not have done time anyway.

B

Yeah there's a deadline of november 11th on this. Isn't that.

A

Yeah, the the deadline is coming pretty soon. um We migrated all the other hem charts. um The only one that we haven't migrated are decks and old proxy and we don't need them anymore, because those were used for the for the for the election last year and we decided to not reuse them this year. So I just have to remove them from the cluster.

A

um The next point that I briefly mentioned, which is the docker hub issue, so the deadline is coming. um Apparently they announced that they will delay the policy, uh the retention policy for a few months, um but we are still looking for alternatives.

A

I I I received the news from um italka hub, so we may um go down the path about paying the subscription on but yeah, otherwise, nothing really major here. um The next topic is about docker images for windows. I think garrett. You have some news here that maybe you want to present. I mentioned.

C

Yeah sure so um it's one of the kind of decisions that came out of last week was that we should be locking to the latest ltsc um release for these, so rather than going with the 1809 1909 and so on, releases go with uh 2019 ltsc.

C

um So we have added kind of like a test phase into the pack of builds to validate that we can actually build um docker containers matching that version, which does seem to work nicely. Well, it works nicely for azure. I am working on the aws piece now and then it's a question of whether or not we need to build our own version of the adopt open, jdk.

C

Image or whether that pr will get merged on the adopt open, jdk repo to actually um produce 2019, ltsc versions.

A

Thanks just for the context, so basically the challenge that we are facing here is: if you want to build a windows, docker image, you need to use um the docker image version for windows need to match the the host machine.

A

So we cannot. So that's that's the reason why we want us to to select one specific version, so we don't maintain every windows um version. um That's why we decided to go with the ltsc 2019.

C

Yeah, so, even even though the ltsc will be patched, um it shouldn't update that version number. It should be the same version of the the kernel really um so as long as windows updates are applied um to the base images, then that should be fine. Okay, perfect thanks.

A

um So, what's what's the current states to the specific tickets? So what are the next step on that topic? Basically, for you, because I detect open gjk, uh I saw pr from slalom from alex on the other gdk projects, but I don't have the feeling that I mean we are kind of blocked there. um We don't have any control on that project.

C

So if we don't get any, um I suppose like yeah adoption on that, getting that one merged we, I think the best option is for us to produce our own image based on that version.

B

We're going to link to the pr yeah, that's that's something yeah.

C

I'll I'll, let that in.

C

D

So the the create our own image install adopt. Oh good, you provided us a link, excellent thanks, gareth.

D

It's not true great.

A

I'm sorry about that. I have a good watchdog.

A

um So, yes, if you, if you could just put a link to that.gdpr, that would be awesome and you did that's really great um and to the gratitude that would be even better.

C

The party right now.

A

Next topic, um any question on the windows occur. Wait no!

A

So the next topic is about the artifactory. um As far as I know, there is no news there. Here uh we are still. I I think that the current status um they are happy with the current situation. Yep mark you have more.

D

Information yeah, so so they were, they were they've started a new set of monitoring based on the change that daniel made okay, the change that daniel made was to switch off the allure use of our repository as a tool. Installer source they've asked the second question: oh hey: what can we do to reduce the volume of data that you're currently storing and the volume was like on the order of three terabytes, so three or four terabytes? So it's a large large repository daniel replied. Yes, we know we need to do that.

D

We have not done that yet. So we have an action yet to do, which is look at, which are the things that are using the most space and identify. If we can do something that does not jeopardize the project and still can reduce our space usage, for instance, the incrementals is one place that we expect we'll find some savings we could make by just declaring a retirement policy for incremental builds.

D

That are, you know more than a few months old, something like that doesn't make sense to keep them forever, anyway. Okay, so let me make a note here that.

D

D

Okay, so that that's a further need, um so we're not done yet, and the action is on us right now and not on jfrog.

A

D

A

Thanks um next topic is about basically the jira testing that that is happening this week. So yesterday the linux foundation gave us access to a public jira instance. um In order to do the test, it's almost the same, except that the endpoint is obviously different. um It's an old, I mean it's. um The data is something like two weeks old because of you. I mean we back up and restore that data. We are still looking at um any blocker for the transition.

A

As far as I'm aware of we are tracking all the information in the google doc, but we haven't identified a major issues. um Do you want me to to review them mark the different issues that you noticed? I mean.

D

Those are actually small.

A

Those are more issues.

D

But I can just drag them in here and we can look at them together. Briefly, we've got tim here and he he detected one that I had missed. uh So if it's okay, let's.

D

Yeah, that's oh good! So let's are you? Okay, olivier if I will just walk us through this, so tim you've found some things that need updates and that's makes sense.

A

So those those upset um those plugins were not available on the testing instance. So I'm just wondering where team found them.

B

That is on the test. Well, the instance that we logged into that linux foundation, url.

A

Okay: okay, because I I I went to the same place and they were not listed.

B

And were there when I first hit the screen show I can see if they're still there, okay.

A

So yeah, that's basically that those plugins um in this case we are using two plugins capture for jira, which can be deleted, so we don't need them that anymore. I have got confirmation from um arnold and suggestimate is, I think, still useful. Basically, what it does is when you, when you create an issue um you try to detect. If someone already created the same issue, um so it suggests other similar issues which is useful um yeah from time to time.

A

um We renew, so I think we just have to push the button update um in this case. If you look at the the widget the dashboard, something that I found weird is indeed some. Some some gadget cannot be reused like this one. This gadget cannot be displayed, but if you delete it, you can recreate it and it work uh correctly. So I have no idea why this specific one is broken.

A

um There are specific, I mean there are few gadgets broken on different dashboards, but again I couldn't identify why it was working this way, because I could delete and recreate them. um If you go down again because I listed a few things, if you go up um in the comment that I did at the beginning,.

C

A

It looks like I cleared that comment. It can summon more if you go up if you, if you go up on the document, if you go up again again again again again before team jack, yeah, just good on testing notes, I put a bunch of comments.

A

um So the different thing that identified that I'm wondering is what about the way we send emails. So right now we are using sangrit to send the emails um that is working correctly, but if we switch the linux foundation, you may have to to use something else. uh Something important to understand. Is we enable dmarc on our domain? So if we want to change the I mean, if we want to allow the linux foundation to send emails, then we also have to update the dns. So that's something to keep in mind.

A

So it's not as as easy. Otherwise we don't send a lot of emails, so the basic same crit account that you have is way enough, but yeah.

A

The second point that I couldn't verify is I shared some credentials with a linux foundation, so they could create a let's encrypt certificate using the dns method and they should only have access to specific text records, um I'm still wondering if it's working for them, because in this case we are not using issues that you can see at org. um So this is something that I would like to validate and finally, something that also, I think we should change, is by default. In the current settings we use jenkins jenkinscia.org domain and the default.

A

One is jenkins that I o, so we should also yes, I thought I thought that this one was the broken one. Yes, it's a broken one, because um you have to configure that inside jiro. So at the moment we have two http proxy in front of it issues uh jenkins.joe and jenkinsia.org, but the jk.org is a duplicated domain. So now the default one jake is that io and we keep taking c.orc for backward compatibility.

A

So, ideally, we should switch to jake in zelda and basically, what happened is um several years ago, someone made a contribution to the project to enable jenkins that are your domain in front of jira, but it never really worked, and so that's why, when you go to issues that you know there are some broken links and stuff like that, because chira doesn't like you to use different domain than the one configured internally.

A

D

So that's the the issues that jenkins started: oops issues at jenkins that I o domain. It's a single global configuration change that we make on jira and it should then work.

A

As expected, yes, so, basically, if you go to the jira configuration, you can specify the domain that you want to use and you have also, to put I mean if you have an http proxy in front of it, your http proxy need to accept that domain, so that that that does that and the thing is we never? We never really update the turret configuration okay, so.

D

So then, this one is one we would need them to to agree that they will change the default domain, or is that something that's in our control? We could change it after the migration.

A

Is complete so the in our case? The thing is, I have that's what the reason why I never change. That is because um the domain is hardcoded in the tomcat configuration in our case, it's defined in the jira database, and we have an http proxy. So the reason why we never really officially switched to issues such as because we never took the time to clean up that and to be sure that um it's working as expected.

D

Okay, so so this really is something where we can coordinate it with anton and make that change, but we really can't prove it works until final final transition.

A

The week of november 9., but yeah in this case in this case, because they provided a different um url, uh because it's a linux foundation endpoints, um then they can just put whatever they want there, okay and otherwise. I just quickly documented the different ways that I test the new instance. I haven't identified any major blocker here so to me, I think we should uh plan for the next step.

A

Great excellent.

D

Oh and I need to include a link to that- google doc that google doc is intentionally not publicly visit public readable because I thought, if we someone one of us, is testing needed to say something in there. That was not for public consumption.

D

So if you're, not a member, not on that doc- and you want to see it just click to ask for permissions.

A

Yeah, so basically that's the way we work when we don't want to be public, I mean for obvious reason. Then we usually ask the person to manifest this interest, and then we just share access, so we don't want to be private. It's just that. We want to know who has access to the information. Basically.

D

Before we before, we leave that topic, everyone's still, okay with the week of november 9th as the transition week for final transition. I assume we continue testing for a few more days here, hoping not to find a blocker. If we don't find a blocker, we go with november.

A

9. is that, okay to me, that's perfect, because sooner is better with jira anyway. um This basically gives us the next week to react whatever happened, and then we can plan um a down downtime.

A

We still have to look with the linux foundation, how much time it will take to backup the current during the the current jira instance and restore the backup, because, basically, we must be sure that nobody updates uh the prediction which your instance, while we are backuping and restoring so the g-rat instance will probably be done for a few days, a few hours, sorry right, yeah at least a few hours at least a few hours.

A

So we try to communicate about what will be done next week once we synchronize with the linux foundation. Great. The next topic is about azure credit. So the first thing that we have to um we, we are still getting warning that the bill hasn't been paid. um We I'm still waiting for news from the linux foundation, um but yeah, that's the current states. Otherwise, if I understand correctly, you were able to to introduce a request to get credit from microsoft. Mark.

D

Last week, yeah well kayla linville. Our customer success manager at microsoft has submitted a request to her management, asking that we be granted credit credits, and I don't know if they will say yes say no say some. I assumed any credit that microsoft's willing to give us will help offset the cost that cdf is currently paying for our azure resources. So I thought if they say yes to anything, it's a win.

A

Yep, that would be really awesome because yeah, we are not moving away from azure anyway,.

A

Any other topic that you want to highlight and for me because then I think we can close the meeting earlier. It doesn't make sense to keep it for 30 minutes if we don't have any more things to say. So, thanks for your time and see you on rc thanks everybody.

A