Jenkins Infrastructure Project, 25 May 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: 2021 05 25 Jenkins Infra Meeting

Description

Jenkins infrastructure meeting May 25, 2021, including a demonstration and discussion of the open telemetry plugin and how it improves observability of Jenkins controllers, pipelines, and jobs

A

Hi everybody welcome for this new jenkins infrastructure meeting and today we have quite a few topic and we also have victor who joined to do um a small demo. But first, let's look at a few announcements that we have. The first thing is we just noticed issues with the windows package for the latest weekly release to the 294..

A

For some reason, we can't um start the container that package windows, so that really is a big delay, but otherwise packages are available for the other distribution as debian red at ncos. The second thing that I want to announce is: we now have access to a discourse account and that service is available on community.jenkins that I own at the moment. We don't want to open it for everybody, so it's still in a beta mode. We are looking for people who can help us to configure it. So we really.

A

We would like to use that that service to help um and organize questions around jenkins, so the first focus will be about the jenkins user community and then we'll open it to a broader group of people. So if you are interested to help just drop me, a message with your gmail account and you'll have an invite.

A

And finally, the third major issue that I want to share we've been a little bit behind with this one. So we noticed last week that a puppet certificate expired a few months ago, and so, while we fixed the puppet master and few puppet agents, we now have to to go back on every on every machine to be sure that puppet is in sync, um we'll have a little bit more, we'll have more time in the coming weeks to fix that.

A

So, let's start with the first topic. So since we have victor who joined us today to do to present um how we can help with the open, telemetry plugin or on jenkins instances, I'm going to switch this topic at um the first one so based on discussion, we have had we had with this victor and other people at elastic, um so victor has been working on the open, telemetry, plugin and so ways to visualize information.

A

So, in this case, it's not about monitoring jenkins to know if jenkins is working or not, but it's more how we can collect information and visualize information. We will visualize those information to detect um wrong behaviors. um Maybe um victor you want to share a little bit here.

A

Sure I'm going to start the screen. First yeah. Let me show that. Let me stop.

A

A

So while you prepare the sharing, um our objective is to is to to to use ci touching kenzario um the infrared ci as a way to test this and yeah. Can you share.

A

B

Here we are uh right. The very first thing I would like to introduce is the plugin it's already available in the yankee ci, or has been released a few versions already. We are in the 0.x branch at the moment, so it's not yet 1.0 for the yeah. Basically, we need to standardize all the name. Information to be sure like they can be used, uh won't change in the future. For such, we need to engage with different communities. You know the telemetry spectrum and also in the uh continuously foundation, the elevator of context.

B

The idea of this plugin is to enable distributed traces for every single bill that happens in jenkins, so you support different type of jobs. Freestyle um pipelines, uh though the more information you gather is, will be always to the pipelines.

B

So we just decided to focus more on the pipelines itself and the point is for every single build. There will be a particular transaction if created as an apm distributed trace. So this particular plugin is completely agnostic. It does support different vendors uh because the use open telemetry is becoming more like a standard uh from anything related with logs, metrics and traces. um So that's why we took the lead to use open telemetry to be completely vendor agnostic.

B

So, from the architecture point of view you can plug to plug this plugin for any kind of backend, so the supported ones at the moment is any, but you can customize it as you wish, but let's say you can plug to jager, promote the use elasticsearch and the visualization as well. You can find which one could be in on it could be as well in kibana.

B

So the idea is you install the plugin, and then you start collecting data such as every issue we trace for every bill and metrics as well. Lots will become short sooner than later. So a little bit more of context. uh All the information we are gathering from the transaction point of view from the build point of view is the name of the job, the type of the job, whether it's a multi branch type, holland that it took any description.

B

The whole number you know, traditional data that normally is in the middle of every bill for the spans are normally related to.

B

If you do a key check out in a pipeline or you do a shell step, so every single built-in step either provided by the core of ganking so or any plugin that you use it's gonna, be basically a report as a spam uh with all the details, such as what is the plugin version, what is the name of the plugin?

B

That's for that particular step, the name of the step you use any labels, such as for cell steps or batch or powershell they will they will be populated as well anything related with the checkout of the source control such as the repo name of the branch, the user. This sort of things will be also information as attributes, and there are other things such as metrics the number of bills, the number of failures. How big is the queue? How many items has been left the queue?

B

That is, a space related with the garbage collector and system cpu? So it has been added recently, so a layer is more of context as well. uh The only thing that you need to do to install this plugin is basically either you use, gcax, sorry, decals or using the ui interface, which basically, you specify the entry point. uh What is the credential that you need to use from the ui point of view, which is the dashboard that you would like to use now?

B

A couple of examples: how you look fly from the user interface for every single wheel? There will be always a link you can see in this particular pinky box uh with the link, and this is customized, so you have your own iphone. You won't see any reference to, as this is just an example.

B

uh You can even customize this particular description. um Then it will directly link to the distributed trains. So this is probably the most dummy example of running a bill is just check out a source control uh run maven for building and packaging. So what happens is the entire transaction? This is the name it took 10 seconds and then for every stage and every step you will start seeing spawns.

B

So how long did it take to provision a machine in this case just less than a second or because it was a local machine?

B

The time they took to check out the source code took one second, the time to build on package from scratch to four or five seconds, and then this is purely our echo.

B

This visualization is specific for relaxing you have different ones for and zipkin uh all of them, because it's open, telemetry based but yeah, so everything related with open television tagging is here, feel free to use. uh Ask any question and anything about what is the roadmap or other things that we are planning to do is also in the discussion on any issues, but in the project you can see. Where are the different topics?

B

We are interested to move further, which are the one that we are postponing so yeah, so it's public and you can access anyway. So a little bit more context um about distributed. Traces probably is more used for any application, uh so just to monitor a real production environment, every transaction or everything I interact with any microservices or third party services. People use these particular uh receiving traces, so we thought we could apply the same for jenkins, uh more specifically to troubleshoot and analyze how to analyze when something goes bad in instability with the ci ecosystem.

B

So if I want to give you a demo, uh we already have a couple of ganking instances, the one I would like to show you probably is the one related to our production instance at the moment. So this is one of the instances that it's published accessible anyway. It's similar related with the atm presently in the elastic organization, I'm going to log in and then from there.

B

I don't want to go deeper in details, I'm going to click a couple of folders to show you one of probably of the project that I'm more interested to show you, because it's a real life demo, uh please bear with me as well. So this is a multi-job project, um so this is the cell library we have and it's called again public accessible.

B

So in this case, as I mentioned earlier, there is a link. The link is here, and we can click on that and we can visualize the latest bill from the distributed traces, how it looks like so I'm going to open that one.

B

Meanwhile, I'm going to go back to the previous tab and I'm going to open uh bill, one of them for the previous one like just finished, and I want to see how is the blue ocean view and then you, you get more like one to one, how it looks from the wheels and how it looks uh from the dc with the traces. So this is quite linear. It's just probably the most traditional steps, like you check out source code.

B

You know some linking gives you some tests and other things, but in theory it's more like what basically is a global java project, but it's just sequence of running every every from the blues and every every step that you see here is most likely a step in the pipeline. That does something specifically. This is a.

C

B

This is validating if the machine is a linux or not, then loading a file right. So it's quite precisely. Well, it's quite accurate what it does every single step, so in the ui from the blue ocean. We see this. How does it look like in the atm display traces?

B

So this is one of the environments we have um so in this case, um this certain granularity, because obviously the ui view here is more collapse compared to this one, even though we can collapse, so we can just focus on the different stages and then we can move forward. Probably will be easy. So in this case, what we have here is this entire view of a distributed trace, and this is a transaction, so this particular build or job was execute.

B

So you can see here certain metadata, such as. What's the name of the pipeline, I don't know, I'm gonna zoom a little bit more uh was whether it's a multi-branch type another, since the time it took and the url.

B

So this is the information regarding the transaction, then, for every single stage we are going to see the same way. We have the checkout, which is the default from the scm. Sorry, from the declarative pipeline.

B

The different specific one we have is checkout, we check licenses, so you can see like this stage here are the one if I click back the idea, so those are spans and if you go a little bit deeper or let me go deeper, for instance, and the checkout happens, and by default we already have a sleep, but that's something specific for us anyway.

B

But what is important here is from checkout. We can visualize for the details right by and what is your repository? uh What's the branch? uh What is it protocol that has been used and if it's an ssh or not? So these are the standard names. Sorry attributes we are using, others need to be standardized and that's what I meant earlier.

B

So every single step that happens in the pipeline is a span and there are meta standards or methods. Panels check out the stages which are meta spawns the similar for how long it took to provision a machine for this. One is an ephemeral one and it took almost two minutes since the request of the worker was asked by the pioneer. So this is idea, so every single span you see here will give you a sense of what's going on right.

B

We don't see any failures in this case and everything works out of the box, but then you can go deeper to any metadata or any locks regarding the build or the machines that has been used. This is not yet integrated much, but in the future you will be um other things that are important is how you can visualize metrics how you can visualize what's going on in your ci.

B

So there are a couple of dashboards that give you a sense of the healthy of your system and all of them are based on distributed traces rather than metrics.

B

At the moment, all the metrics we gather from the plugin are not used much, but probably uh those are another services that I want to sell. I want to go back to the open telemetry and I went back to the dashboard, so there are a couple of them. I'm going to click on the provisioner and I'm going to click on lcic overview, so provisionary is about the build queue.

B

How many machine has been provisioned and how long that it took how how much time spent so it's trying to for every time you run a pipeline and there is a node request. You will see an entry key and the timing. That's ideal gathering detail from these particular transactions funds and show these details. So in this case, in the last 50 minutes has been requested at once at this particular peak 11 workers, and then we can see the number of jobs that are queueing at the moment.

B

So those are information, like you know, in sense of what's going on.

B

If we move to the traditional dashboards, that probably you made with other kind of tools, so we have a similar one uh where we monitor the queue we monitor the status of the jobs, how they look like this is in the last 15 minutes, the number of workers that has been requested, the number of agents that are alive at that particular time uh all the things that are more precisely about the biking, the number of steps, how often they are called the time it takes in duration- is in nanoseconds or milliseconds, and what is the top 10 of jobs that has been executed and the number of steps steps that are triggered per minute shorter.

B

If we have any failures, we'll see here as well, failures such as traditional javascript traces, like the python provider, there is a connection issue and so much so this is some of the dashboards that are already public as well in the behavior, so you can consume that as you wish.

B

um So there is a lot of room here to do.

B

We are in the transition of using this in our production instances working with the community to to help them to use this and gathering use cases or valley scenarios that are important to show a lot of things different here. We are trying to prioritize this uh yeah emotionally. This is how it looks at the moment. um I don't know if you have any more accurate questions that you want me to answer, but in a brief summary, this is probably the demo. I wanted to be.

B

I I don't listen to you at the moment.

A

You have a sound issue.

A

I was muted sorry um thanks victor for the demo, that's really impressive. um What is interesting here is, while monitoring jenkins is pretty easy. I mean you know if jenkins is running or it is not, but really having information about how it behaves. um If it can provision nodes, how long it takes to provision nodes with which protocol is used and so on. That would be really interesting, especially in the on ci jenkins that I o, because we regularly switch between um cloud environment.

A

We use different, I mean we regularly test um latest plugins and so on, so that would definitely benefits to the jenkins community.

A

So I'm really excited to see this so another interesting, I think, um area that we could work is to identify what are the different scenarios, um because I'm sure other people have other questions as well, and so, if we can build generic dashboards, that people can can get inspired, that would be really nice to drive.

A

So I think what we could try is to let's say, focus why I mean once a month something like that to to to to report a little bit the improvement we did here and the kind of question that we were able to answer. That would be really nice.

A

um Another thing um I don't think you're using the plugin matrix the metric plugin.

A

So I don't think that you're relying on the player, plugin metric.

B

A

B

At all, so this particular information we are gathering is purely in the open, telemetry spectrum. So all the all the information we are gathering and all the sdks we are using to create these distributed traces is purely open telemetry.

B

So all the information I mentioned about the java virtual machine garbage collector and so on- he's not using the metrics but he's using the open telemetry that provides under the hood from the other flag.

B

But again this is pretty much new. We just merged the pr. A couple of days ago.

A

Yeah, this is a really nice project and I'm really excited to work on that. um So thanks thanks victor for your for that. For that demo, any question.

A

No so lucky for you lucky for you. It was recorded so yeah.

C

I missed something, but I have a few good news from the cdf lens. So when you finish the management that you can discuss these topics, if any time later,.

A

Okay, um so then I propose to quickly move on this course, so I sent an email this morning to collect uh feedback so, as I said, we deployed so the company behind this course offered the project to sponsor uh with the business here. So we have access to quite a lot of things. The service is available on community that jenkins rayo. At the moment, it only work on inviting me as we want to better understand the tool, so if you are interested to participate, feel free to reach out.

A

um So for me, the main question, the the main question that I have to solve in the coming days is how we authenticate with that tool. Do we only rely on github sso? um The benefit that I have about using databases? So is we don't rely on the third service and everybody who contributed to jkins already has github accounts, and the other question is how do we also allow people to use their jenkins account to connect on this course, because it's also not a possibility.

C

Well, I recommend it in the mailing list, but actually my proposal would be to focus on the linux foundation sso, if possible, uh because yeah, we definitely don't want to keep our hold up for long term as we discussed in previous meetings. So we want to have it, but not for every jenkins user and ones who use jira et cetera, but for people who don't need permissions to course systems.

C

So for me my preference would be to not use github directly but to use linux foundation, because in this case you get a also support for gmail g suite accounts and other things. uh Not everyone who would be using this course would have a github account.

C

Well, I think it's not to be able to create that unless you're based in a country where you cannot use a github account, for example, if you're based in crimea in iran etc. Well, iran is no longer concerned, but yeah overall having a foundation level account would be nice.

A

Yeah that that's a good point here that you mentioned thinking's user and jenkins developers.

C

Yeah so again it's preference. I have no idea about technical feasibility and the guess. Android greenberg will eventually comment on the thread and see whether I'm completely nuts or not, but yeah.

A

So last time we had a discussion about using linux foundation, authentication system that was okay, I'm not sure if we have access to the groups but yeah. This is some. Those are technical implementations that we can, that I have that. Are that have to see with andrew.

C

But yeah this is our importance, but we do not really want to use genki sold up for new systems and services we deployed unless there is benefit in that.

A

Okay, so we kind of have an agreement that will try to avoid the changes accounts.

A

um Thank you, and the other I mean element that I would like to clarify is: when do we consider that the discourse instance is ready to be used broadly? What would be the acceptance criteria for this.

C

I think for that we need some group of champions who would drive that option, and this group is already performing in the developer manifest from what I can tell.

A

C

Then yeah, once these things are removed, once everything is evaluated, there can be just a proposal that.

C

Then we just apply common process if everyone, if there is a consensus, if we hold for that okay, it's official, I wouldn't really want to invent a wheel there, so it might be jeff, etc. If you want to write specifications for that, but yeah, I think the main point for us is to just evolve this system and see whether it fits our needs.

A

Sounds we have an agreement there, um that's perfect, so we cover all the points that I have regarding this course. So I propose to move to the next epic. That's a leg want to bring here. Cdf news.

C

Oh okay, uh so yeah one of the cdf. You said that uh today I brought up a question about uh transferring kws account so.

A

C

There is no concern from the technical oversight committee and from racing tracy is supportive about it. Tracy took on it like to actually explore what would it take, because we know that spinnaker already has aws account, but if spinnaker account and jenkins account, you will be in the same situation like call this and jenkins account in principle in terms of billing. So we need to figure it out and well. Tracy will help us with that. There was no opposition and yep.

C

I will be able to drive it on the cdft, it's kind of clear to be announced, but I will join the technical oversight committee because we were only four naming nations for four seats, so there will be no elections.

C

This year so yeah effective july 1st, I will be stiff top number, basically replacing the kiki in this role.

A

Which sounds really good news.

C

So what else yeah nothing else related to cdf at the moment.

A

So we don't, we don't have we don't have any estimated time for the ios accounts right.

C

Yeah no estimated time and yeah, we agree on all sides. So what is currently happening is what I sent to the information list. We are splitting the cloud visa account so that there would be independent jenkins account to which you will connect all sponsorships, so this account will be ready for transition will be technically able to add more contributors at this stage.

C

um I believe so, but it won't be owned by cdf. So there will be somebody who is formerly the owner, maybe you'll be still club b's. Maybe the sum of individual.

A

C

If they interest from that, I'm not uh but yeah it's yet to be decided, but we are doing the initial transition stages and then federal. We initiated the discussion with the cdf but yeah. Let's set the expectation.

C

If we get it done by autumn, it will be already an achievement, so we don't expect it to happen overnight.

A

Yeah that I mean that's regret, because then that means that we would be able to invite um to that account and then include these employees, which is really nice.

C

But we should be able to once the cloud-based migration will complete and thanks to band welding thanks to ray sullivan and other public employees that is being processed quite quickly. So there is spring planning and other things apply, but generally it should be done soon.

A

That's awesome. We are running over time for this meeting. So um do you have any other topic that you want to bring here so so yeah then thank you for your time and I'll see you later on rc goodbye and thanks victor and even thanks. Everyone.