Jenkins Infrastructure Project, 15 Sep 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: 2020 09 15 Jenkins Infra Meeting

Description

Jenkins infrastructure team meeting September 15, 2020. Topics included incrementals, Kubernetes upgrade, mirror status report, JFrog artifactory, Jira upgrade plans, and Oracle Cloud

A

Welcome everyone: this is the jenkins infrastructure meeting it's september, 15th 2020.. uh Let's take a look at the agenda and be sure that we've got the right things and talk about them in the right order. So what I had on the agenda.

A

Incrementals status report tim from you just briefly: kubernetes upgrade mirror status report if anything uh docker terms of service, not sure if we need that one uh jfrog artifactory, that's a heads up, one that I wanted to talk briefly about jira, upgrade status, oracle, cloud, azure, customer success and then one that I'd like to propose to move earlier in the meeting. To be sure we get to it release status reports, anything else that people are feeling like. We need to include on the agenda.

B

A

Sorry for the long agenda.

A

Okay, all right, then uh tim, you want to give us the latest on incrementals on kubernetes.

B

uh There's been a couple of small features.

C

uh From jesse and gavin and all deployed fine, no issues and no no reports, many issues with that.

A

Excellent, so that means we've really switched away from the azure functions entirely to running on on a kubernetes-based docker image.

D

For this there are still other functions that we have on azure. ah There are okay, yeah.

C

Anything that does anything.

D

um There are things that are being hit um pretty frequently. If I remember, let me look real quick.

C

There's a function for promoting staging to production automatically, but that's been broken for ages. It's probably pretty simple to fix, but.

D

Sorry, you guys can you can continue. I have to enter my code and stuff from my authenticator. So that's going to be amazing.

A

Well, but I think that's a that's an interesting one to to have on the agenda, even if we don't get to until next week. Great to be sure that if there are other azure functions, we could we could or should migrate away from it's good to know.

C

There's no reason to migrate away. It was the reason that we flew in korean tools was mostly that some of the incremental runs take two take longer than you're allowed to run on the function. We were doing it on and also the log logs weren't, the easiest and weren't the most accessible.

D

And since incremental is this used so much? It was just a no-brainer to kind of move it over and on to something. That's a little bit more stable and under our control.

A

Great okay, so so this this migration was uh an intentional because of demands we place on the function, not some general purpose limitation of azure functions. We should move away from thanks for the academy.

D

There's so the only other one that's getting hit somewhat frequently is comment, logger, which I am not sure what it's actually doing. I haven't looked at the code.

C

It posts some of the jenkins build result into the github pr, at least, and I think it only runs on the emperor orc. It's.

D

Probably both king jenkins.

C

He's probably posting gems to I o build failures, just.

D

If it still works, I haven't seen it at all. Okay,.

C

It probably hasn't worked since the password reset. To be honest, I don't think I've seen that in ages.

D

Yeah so so we have an analytics bouncer. We have comment logger we have evergreen upload nevergreen is dead, so we can get rid of that one. um Then the infra, auto pr or the um four that are on there.

C

Yeah, so I think those two just need fixing probably would be good to get them back, especially that infrared.

A

Okay, so so I'll have to I'll listen to the to the recording later and put the detailed list, and I wasn't really fast enough to get the entire list into the notes alex.

D

I I think we can just say, look at the four remaining functions and see if we actually need.

C

E

D

Working with adam for a while so they're, obviously not critical to our infrastructure. At this point right.

A

D

A

D

No sorry, I was done.

B

Kubernetes upgrade, let me just check if the one issue is still.

B

C

Actually that came right um so that yeah that was yeah that was done. I did it. I was gonna. Do it the day afternoon for a meeting, then I realized it was lts release day. So I should probably not do that um so the day after that um went from 115 to so I had to be a two stage, so I did it 115 to 116 and then 116 to 117.

C

It was actually four stage because you have to do the masters and then the node pulls um and then wait for all the notepals to finish before you do the next one um but yeah. So it was all fine. The only issue I had was.

C

uh Loki was still bound to an exclusive volume and it wouldn't let go. I ended up just leaving it because I think it would probably come right um and I just checked it now and it has it. It was and nothing needed changing I and then I ran the build and it all was fine.

C

Excellent, so this is.

A

C

Yeah yeah I've. I also found that the jenkins, so the terraform code was still pointing to 1.14 so olivia we did. The last update didn't update terraform, um but I've updated that to the latest version.

A

Excellent, so, and and really excellent, because I hadn't I did not detect one single thing relative to that. That's great. I assume there was some some brief offline periods for some pieces of this.

C

Yeah some alerts did trigger. It was about under a minutes downtime. I think I don't know why it happened. It shouldn't really have happened, but um yeah there was like a minute blood. Oh.

A

Well, excellent, that's great.

A

Anything more that needs to be done there, anything that you want to flag.

E

No, I wasn't fine.

A

Excellent, thank you. Mirror status report.

C

uh I haven't any time to work on the decommissioning, but we just need to be commissioner: decommission the old one.

A

When we decommission.

C

Wait: yeah the old server, so it's we need a. We need to find the latest windows download link and um get the and move the lts rc process to release.ci.

C

And I notice right, my chrome is flagging warnings um on the um windows download because it's a mixed content, so it's it's currently pointing at the old one which is on http and by default it won't. Let me download it.

A

Okay, yeah and that that makes sense that it would flag it right because you're you're switching from secure to insecure, okay.

E

Yeah, so that's just a needs fixing there might already be a link, I'm not sure.

E

Well, I'm not sure that the the builds.

A

Are being uploaded.

E

Yeah, we need to figure that out.

C

A

Thank you thanks very much uh change in docker terms of service. I have no nothing new to report there compared to last week, any any red flags from.

D

D

I haven't been following it very much at this point.

C

It was time skip.

A

Okay, great jfrog artifactory, so there's been discussion, so jfrog hosts repos.jenkins.hci.org for us and they they have flagged to us. Hey we're worried that you're using an awful lot of bandwidth and an awful lot of uh yeah bandwidth was the primary concern. They've scheduled a meeting today with me, daniel beck and several of their people to discuss we'll, listen, hear what their concerns are. Try to understand them.

A

Our goal is to continue hosting our artifacts on that repository and if they say, oh, you need to reduce in this way or that way, we'll look for ways to do those reductions.

A

I'd really rather not have to create a project that moves us off, jfrog artifactory to some other artifact hosting system. I just don't want to spend the time, uh I'm assuming that the two of you are okay, not being included in that meeting. We could certainly include you if you feel to join it will be. It will start in about three or four hours, I'm fine, not being there.

C

Can't do it anyway, but no I'm fine.

A

Okay, all right, okay, last topic or no next topic, release status report so 2.257 released today and in an awkward, embarrassing thing. It had the same problem as 2.256, no microsoft, no msi installer and the reason it has no msi installer is microsoft. Minor numbers and major numbers cannot exceed 256., so we had the same failure last week and I failed to detect it sorry about that. um We won't get a new msi installer until we find a different way to represent the version number in that installer.

A

There we go 256.

C

A

Number yeah, I was thinking it's 2.255 dot, build number right, so 2.255.

A

dot, 257 and then it becomes 255 at 258..

D

What about uh lts.

A

uh Lta fourth, a fourth component is allowed, so how many of that two.

D

How many bits that are available? I've ever had more than like four out for lts in a single version, but.

A

As far as I know, 16 bits in the third position, so this position uh is, is a 16-bit quantity and the fourth position they described as being ignored and having no limit.

D

So that the problem, though, is if it's possibly ignored, then, if you try and update from one lts dot release to another, it may not work.

A

uh That's and that's a valid point.

D

A

I believe that.

D

In lts's we're actually I'm actually using that third component for the lts portion.

A

ah Okay, so, and- and I think that's pro alex- what I would lobby is- let's take that into this issue that I raised so that we've got an authoritative location to discuss it, because it's a valid point. The the general upgrade process that I see most jenkins, admins users is they install the msi once and then they use the inside jenkins um upgrade to the newest version. So just as a war upgrade rather than running the msi again and doing an msi based upgrade.

A

So so, but it's a valid point: it's how do we handle lts releases if we shift right.

A

And I'm open to other suggestions.

A

Certainly I I I'm hesitant one that I thought it was well. We could switch to jenkins three, but I'm not sure that the project is ready for us to declare jenkins three linux did it. They just said this kernel is now such and such without any major actual changes, but I suspect that will generate more noise in the in the community than we're really ready to do just because of a windows of the windows limitation.

C

22, just like a two on the end of the first first digit.

A

Oh, oh interesting, were you 22., so your suggestion was: do this 22.2? No, but that still doesn't help us? Oh, oh, oh, and then we would decrement this to be 57. Is that what you're thinking shift left? There's 22.0.

A

uh Yeah, okay yeah, so so my proposal is: let's have the discussion in this in this um in the issue and yeah. That's an interesting alternative.

A

2.255.257, but that then has the lts problem that you noted alex.

A

We could go 22 there, because there is, there is a concept of a display version mentioned yeah.

D

We're using that right now, oh we're already using that yeah, that's we're using it for the lts right now, good, I'm looking at I'm looking at from looking at the um the wix proj file, we wait a sec.

D

Let me look at it and we can discuss the in the bug.

A

Great all right, excellent, okay.

D

As long as we don't don't go above a thousand versions for a s uh s for one uh release line like if we don't go up to two point, you know one thousand. I think we can make it work.

A

Oh good, okay, great.

A

Great okay, so alex you're. Okay! If we have that discussion in this in that yeah in that ticket, that's fine great all right.

A

Anything else on release status reports we we did release last week, 2.2 49.1 and it's been a little more noisy than than a typical lps, but not a lot more noisy part of it was upgrade center changes. Part of it is windows, install changes and surprisingly little noise about ui improvements just been very successful.

A

Next topic, jira upgrade plan here the linux foundation has connected with me. They've sent the poll to schedule our next meeting. I still have not made any progress on the action items I'll do that this week, it's it's getting closer and closer to november 30th, when the support of jira 7.13 ends any questions on jira upgrade.

A

Okay, oracle cloud conversation, so I met with oracle cloud or representatives from workforce cloud two weeks ago, and um they share that they're interested in helping the jenkins project and having a more active jenkins solution available on oracle cloud. They sent me a non-disclosure agreement and offered to review our azure bill.

A

Looking for cost saving opportunities, I'm gonna have to talk to the governance board to see what what I can or cannot do with ndas as an individual, I'm happy to enter into an nda with them, but I can't certainly enter into an nda with them for as the jenkins project or anything like that, uh alex any guidance from you on that. One. In terms of past experience with non-disclosure agreements,.

D

um I haven't had any in terms of as a board member at this point, so um we definitely need to bring it up with the full board, because there are people on the board who have probably had experience with it, like tyler and uh and so forth. So it'd be good to have a discussion with the board for sure.

A

Great I'll raise it as a question in the mailing list today and hope to get it onto the agenda for tomorrow's board meeting all right last topic was: we have an azure customer success manager assigned to us now. Kayla, linville and customer success. Manager in this case is intending to find ways to help us use azure well and effectively.

A

She's provided some documentation that I've I've still got a review and I'll send it out to infrastructure mailing list, as I as I've reviewed it and get some insights what we might do with the information she's provided.

A

Any other topics we need to go over.

A

A

All right, the recording will be available separately. It takes about an hour usually to process it thanks very much we'll end. The meeting here.