Jenkins Cloud Native SIG, 14 Jun 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: ☀️ GSoC CloudEvents plugin Community bonding meeting 2021.6.07

Description

At today's third community bonding meeting for GSoC CloudEvents plugin, we discussed various plugins that are listening to and capturing events in Jenkins pipelines.

A

Great welcome to today's uh jenkins gsoc meeting for the cloud events plugin still community bonding, so you know we're exploring the project and hasn't quite reached the coding phrase, but trujillo has been very engaged. So it's great you can tell us you were mentioning before to me. Some of the work you've been doing this week right the freedom. Yes.

B

um Yeah so this week you know it was not a lot of coding. It was more like discovering the past plugins that we talked about, um so it was the statistics together plug-in.

B

So I had it on my system and I was just sort of playing around with it understanding the code and then I also found similar plug-in for listener implementations, um one of which was the extreme notification plugin, which is a little different, but it has the the same um sort of events that it's emitting, um and then there was another plug-in which I looked into two days earlier, and then oleg also mentioned something which I had seen, which is the um jqs monitoring plug-in, and it's it's a bit similar in the sense that it's also like publishing information that we need about.

B

You know jenkins objects like about jobs, about cue about the builds which are currently in the in the queue, but it's also different, because it's using stapler, which is quiet, I quite get it. You know it's like the url binding that I'm not exactly sure. Why um like how it can help us, because the statistics gatherer plug-in, I think, is it's a really good way.

B

Just the necessary implementation is going to provide us all information so, um but, but you know, that was something that I wanted to talk about with you guys and just understanding stakeholder implementation, um and I also like looked into other listener implementations, and I have like noted down all the possible um like implementations for the events that we can. You know possibly emit uh and yeah it was. It was mostly, you know, just understanding a lot of like jenkins native things as well.

B

Just uh you know, like hudson in general and all the you know, actions and and then there's the root action. Then there's you know um what was that? Actually, I kind of forgot the name of that. um I think it's. The.

B

Any any anyway, um so yeah that has been the work for this week and you know like the coding, mostly involved. um So I was while I was trying the statistics gathered plugin, I'm just trying to understand how how it works. So, just a little bit of you know tweaking the code and then also running.

A

B

And using um have that as a source which is just sending um events out to an http sync uh and then I I wrapped one of that event into like like a cloud event, so it would look like with student cloud event headers um and since that data center has clarified payload, so yeah, um I think you know the next step would be just really going into what, but first obviously understanding stapler.

B

And why could that be useful here? um And you know just moving on from the discussion plugin and just deciding on the sort of events that we need to be editing.

B

If there's any any, you know more configuration that needs to be done. I know the the statistics gather plug-in it's a little um in the ui part, it's a little different because you have all the urls that you can enter, but it does not right off. The bat does not send http requests, so that's into advanced configuration, so you have to click on advanced options and get the uh okay sandy should keep your class and that's when it gets relative, um just understanding those things and maybe perhaps also start talking a bit about um jenkins.

B

As like, async so consuming advancing for that, I've been looking into the the generic trigger which um uses like root action. It has a url um where it gets minded to an object, um and that's where it's like listening on to requests which is being sent by another systems, um but just getting a bit more understanding of how we can extend that and then one from just okay when a when an event comes in the most.

B

You know, the common thing is to trigger a build and have that connected to like a job or a project. That's that gets triggered for the build, but there could be other things too. You know, um maybe if, if um if something is in the queue and if an event comes in, maybe you want to push it out of the queue or something like that. I'm just talking.

C

About so, to uh what extent have you tried the jqs stuff, uh so the jquery stuff that olek mentioned, like I've, never even heard of it, because I don't have as much experience as him, um but a good way to go ahead with this would be to just kind of see which one you know suits you better in terms of like working and uh kind of just like compare, which kind of uh like if listener is better or if, like the jqs stuff, is better.

B

Yeah, so with gqs, what it basically was doing is, um I think I have it on my local, like online jenkins system, so I can do a little bit of breaking.

B

D

B

B

B

B

um So all of these, like whatever it's publishing here, you know here or um our cause of blockage and all of these- um these basically are like the exploited like stapler phonics, um stapler export engine which is going to exist for xml.

B

um I get jqs's purpose and if we want to tie all of this information to a url, and so someone was reading off of the url, they would have this information with them.

B

But what I don't understand is um you know, since we are trying to extract all of this information and you know wrap it into xml or json and then send it over. um I certainly don't.

B

Understand because it's doing exactly like essentially the same thing, you know it has um the the stapler sport engine and it just uses some what those native um the hudson jenkins or classes, and then it extracts information about build on a project and then puts information out as excluded being so that's what every disaster is being just xml on document.

B

B

B

um So this is the on.

B

Implementation.

B

B

So it's basically getting all of this information again from the executor or like whatever is executing it and then using.

B

So you know, I understand that if we want to tie this information to a url and then put it there as xml or even as juice on it, it works with json.

B

I haven't looked at it yet, but um but it works. uh So I think I I'm not very clear on how and why would we need to use- or you know just just as sort of um an alternative method. I don't understand how it's going to work. So you know if you, um if you guys, have any um ideas on this or just have any working information that like that good. But it's not extremely okay.

B

If we can probably drop a message in slack or research more about that, um but for now I think, looking at the the there are also a lot of other plugins that I saw there was another. uh You know. The statistics gather was a really good example and then the extreme notification plugin. But then I was looking at extension points um and then I looked at all the the sort of plugins which have extension fonts for the listener implementation. And then there are quite a few and they have very similar implementations.

B

You know not all of them emit the similar events, but a lot of them are using the same implementation um and the jqs monitoring system, which is using stapler.

B

It also is, you know it is sort of pulling that information and it is putting that out into like you know, outside for people or other systems to access through a url. That's what I think. um So you know what what maybe you if we were using stapler? Maybe it would be designing.

B

You know like something puts information into the url and then there's another class which is pulling information out of it and then um wrapping it inside json or if it's in you know it's natively in json, then just having like putting extra information as a cloud event and then putting in more headers and then sending it over to http to the sync.

C

About stapler, there's not a lot, I know about stapler, but uh from water whatever. I know it feels uh it feels like it's something that is closely related to the descriptor describable model that jenkins uh uses to kind of like map. You know what you have on the url and what you have like underneath uh inside jenkins, so it kind of is like a thing which staples the front end and back in together to kind of have this.

C

You know url to object, class object field and like information flows through that way, so I'm not so sure if it's so it's the way to go. But what I would suggest in this case is what we should do. Is we should kind of make a pros and con pawns list, because we have two very distinct ways in which we can get information here you have this listeners and then, on the other hand, you have these uh this exported information of some sort. uh I feel like in this case.

C

We should like look at the pros and cons like for both of them and which fits best for our use case. I'm not I'm not sure if uh the stapler information maybe works well in like headless scenario, where jenkins ui, I don't know, can jenkins ui, be disabled.

C

So I'm thinking okay, if there is, if there is jenkins file running and if we have our plugin installed on jenkins file runner, would it be possible to you know, use the jqs stuff, I'm not sure about that, but I feel like. If we have for the statistics gather, maybe uh it would possibly be able to use that statistics rather implementation through the jenkins file. Runner, if you so, are you aware of jenkins file runner project.

B

About about what.

C

The jenkins filed on our project, um so imagine you have like a headless jenkins, which just runs uh jobs like or based on. Like you have you based, you have this docker. You have this uh image where you provide your uh jenkins file and the plugins are pre-installed in that image of headless jenkins, and once you give the jenkins file, it runs that jenkins file based on whatever plug-ins, are there in there. So it's like a uh closed-off involvement for of jenkins and it makes jenkins kind of for what you call.

B

It wait, are you talking about like jenkins file, or are you talking about something related to jenkins file.

C

It's a it's called jenkins file, runner.

B

Okay, oh yeah yeah, I yeah I um I'm there. I I thought that jenkins runner is like a different concept. I'm.

C

No jenkins filed runner is a different. uh It's it's a project which was created by oleg and, like some other guys, and it basically makes jenkins uh serverless and even if it might be a little heavy in terms of like as an image, it works very well. uh So if the plugin had to be so, I'm thinking this way if the plugin had to be used in a headless mode, where jenkins is not, is not using the ui anymore and is just taking a jenkins file and running it as it has it.

C

uh It should be able to do all these things uh like send out cloud events. When a file runner like when a file runner runs, it should be able to send out these cloud events to the you know, sync, whichever sync is given at that point right, so I may be thinking too far ahead, but the thing is uh that's why what we need to do is like make a pros and cons list. If the listener is more viable for us, or is the uh jqs implementation more viable? For us, it's called jqs yeah.

C

Yes, so we need to. We need to you know, figure that out first, if there are of hard dependencies on any one of them on the ui, then I I think we we need to recon, reconsider and like use the other tool which is available.

C

So apart from these two, there was something else you mentioned as well right.

B

Yes, um they could plug in.

C

uh Not the plugin itself, but the way in which information is captured.

B

um You know, I think, like these two primarily like the the listener implementation just used in several different ways, because there are a lot of implementation examples or like just listener um classes, and then statements, um but again, like I sort of will understand how stapler is going to work. um Even if it's like you know, if you're talking about headless system, um essentially going to be connected to that ui right.

C

B

If you're using um jenkins a stapler and a send- and it's also adding the overhead of just parsing information from the url with that information- is.

C

Going to exist, are you sure it is uh passing information from the ui or is it kind of intercepting between? uh Like you know, information get getting to the ui like is.

B

C

Like listening on certain like endpoints or something.

B

No, no, not why you are.

B

Because, whatever I have looked about, um stapled for now is just it like is going to be binding. Those objects at a particular url and then um whatever we are passing in you know, suppose we want to custom field to exist at the um url. Slash episode whatever it is, uh you have to you know, mention it okay, this is the information that we want to export and then out of that information, that's going to be another another.

C

Okay, can you check one thing uh for me? Can you check if there is a hard dependency on these uh urls.

B

um On wait on the world.

C

On the urls on the plugin kind of knowing the urls, I don't even know if that's a viable question, because I'm not as good with jenkins as oleg or these other guys might be. But I just I just want you to check if there is such a thing of there being a hard dependency on these urls or from the plugin.

C

If so, then, probably it's better to go with the listener implementation. Because from what up from what you've said, I feel like it's. The the monitoring from the jqs monitoring kind of revolves around urls.

B

Yeah, that's that's what even like, like my understanding, is and when it's uh you know, reading information just to like present it to the ui as like we looked up earlier. um I think it's reading off of that that doc, whatever like the plugin, slash api, slash, xml, that's sort of the native. If, if our plugin is like cloud events plugins all of this information is going to exist at cloud lens, plugins xml and that's with the information that we export from our um like java class.

B

That's where all of this information is going to exist and suppose we had a similar system of presenting this information to to the user interface, um we'll be reading off of off of that api center.

C

They are not, they shouldn't stay, uh they should be, they should come and they should go and uh having something like this is not helpful for us in any way, because we don't need to persist this kind of information unless we have a queue in our play like with us and I'm guessing. That cube would be very small because most of the brokering that we'll do we'll do it outside jenkins, so we sh, we shouldn't, persist this kind of information.

C

If something happens, an event is sent and if an event happens on outside and we are the sink, uh the event is red and that's about it like this won't be persisted anywhere it's, but except in the job, for example, where, uh if an event, if a event cloud event triggers a certain job because we've configured it to be so at that point, the job would just be say something like okay cloud job triggered because of cloud event. So this yeah, this information shouldn't be persisted in.

C

Like a you know, single source of truth kind of a this information shouldn't be persistent.

B

Yeah and like another thing is just actually going to be again just reading off of um the url and um and whenever you know we are like we're reading.

B

Like that's what I think again, I'm not sure if something is right, um this is just me sort of putting words out there in my head, which I think.

C

um I just saw the plugin repo and I just wanted to check. When was the last comment made and I checked: okay, uh let's, let's try to uh okay, maybe something that was released seven or eight years ago. With you know, change in email address is like the last comment made.

C

A

Pretty old school.

C

A stapler we should move with, like some discretion and, like maybe maybe understand, like uh you know, if, like this this, this may be like a good opportunity to learn to like understand how this stuff works, but uh like really think, if we can work with this and like see some like newer implementations that are being done and kind of like work in that way, because there is a reason why maybe uh this same method, this method isn't used in the current implementations of any kind of monitoring or like eventing.

C

That is being done in jenkins.

C

Yeah seven eight years ago, is not very promising.

B

I think I would have to actually make a closing cons list. Just to you know, just understand how this works better. You know I'm kind of curious about just understanding steven and how it works.

B

And I also found.

A

For the next thing, let me just jump in and say: I'm not sure you know, there's an events, um meeting eventsig meeting with the continuous delivery foundation this afternoon in about an hour and a half time. So if you're at all, you know able or inclined I I don't know if today's discussion is necessarily personal to you, but just in general, I I would think that you find uh we'll find the conversation interesting.

A

I can send you the link or I'll put it in our channel for this.

C

uh Even sick today we'll be talking about uh tecton cloud event stuff, so we'll be talking about how to get cloud events, so we are making a new controller for tecton in for cloud events and kind of getting cloud events out of tecton through the new control and we'll be working on. You know making those events into cdf events. So at some point uh we'll do the same thing in jenkins.

C

So techtone is like our testing down right now so like this stuff. So it should be interesting if you want to join.

B

B

um And the second is, you know just thinking about the events itself, what sort of things that we want to embed and things about. um So you know there's the information about the build that um has started it's finished uh and then it's running um then there's like doing steps. Of course you know change from one step to another stuff, and then there is, you know the executor or the jenkins computer.

B

If something is online, if something changed or something failed, uh and then there is information about objections itself, I think some shutdown rejections loaded that those were the two which I kind of found interesting say. Thank you, shut down. Virginia is loaded. How is that different from uh it may be? Just okay, maybe it's like you know, there's like there's a cluster and when you say jenkins, computer offline. So that's just talking about one of those like one of the notes, which is often when we just say like jen can shut down.

B

That just means everything is shut down.

B

And then there's um information about like idol and um jenkins item copied item deleted, renamed, um updated those kind of things and then job job started job finish.

B

So do we, I know like we were talking a bit on the slack about starting off with what's the most um sort of used and um that can have priority.

B

Is there anything that we can add to it or something that we can delete or something that we can just think about? I know that we have not talked about just jenkins computer, that it's offline or it's online or it's unavailable or it is configured, um but I think that's that can be a very important event um and you know just out of like in ci cd systems, if you wanna, if something is failed and if you wanna, you know trigger another action of adding a near um node or something like that. Maybe.

C

I I think to get started uh focusing on like the actual jobs and bills is a uh is a good idea. Oh computers could be something we could do further once we are done with the entire build life cycle, so, starting from, like you know, creating jobs or project to like you know, starting bills and the life cycle of bills, and then the steps between them and then after and then after that we can look at. You, know, computers and the cloud stuff where nodes are provisioned.

C

uh Did you find listeners for them as well.

B

Yeah, like it's a future, it's in the uh the other one like the extreme notification plug-in. um So that is so. That also has quite a lot of implementation, and it was really interesting because they are pretty similar, but they still have. You know like different.

B

It's it's! It's computer listener! That's what it's called.

C

What is it called.

B

C

B

It's extreme notification, plugin I'll drop, a.

A

A

I appreciate the name of this.

D

B

Understanding another one which I did not find as relevant to um what we sort of like the events that we really want to publish. It's like this one is more related to jenkins performance, so.

C

There was a new plug-in which was recently released like it recently started the work on it recently started. um I can't put my finger on.

C

It's called open, something.

C

What's an open source monitoring software, apart from like.

C

C

Yeah, open telemetry plugin, I don't know, I don't know how relevant this is right now, what could be.

C

A

Open telemetry is very interesting, but um that's pretty interesting because my I'm strong, my impression on open telemetry is really for more observability once your application is deployed, whereas we're looking for, I guess in many ways greater observability, even tracing when the pipeline is running and capturing all the events that are happening.

C

Yeah yeah, that is definitely what we are going for. uh I was thinking of this plugin in the uh in a way that we could, you know, just see like what how it is implemented, and maybe if there are some, uh you know a few things if we can find. I don't know, I'm just I'm just uh trying to understand the landscape here and if there are like any kind of monitoring plug-ins in jenkins, uh if it is a monitoring plug-in or like an eventing plug-in.

C

It's then a good example for us to just dive into see how it's implemented and then see. What's the best possible solution, we can come up with uh based on these different examples, something that's worked best in the past or maybe, if there's something we need to figure out.

D

C

Like in yeah like here, there is the computer listener being.

C

C

I think it's more around the you know, understanding performance of jenkins.

D

D

D

A

The jenkins monitoring plugin- I haven't- looked at it, but um yeah.

B

I actually found a really cool other cool plugin by jeff, or he has worked on it and it uses the same implementation.

B

Right it could have auto status plugin. It's pretty.

C

This monitoring plug-in made me think about four keys. For some reason, the four.

D

C

Where you can see all the events going in and out probably from jenkins, can I see like a graph.

C

B

B

Yeah, you know most of these.

B

And like the latest ones, or the some of them are pretty old, but you think most of them are pretty, um you know decent and they are using listener implementation, even if it's on like pipeline and then looking at the github status, I like to give up on status plugin- and this is also using this new implementation like craft. Mr pickling.

B

C

D

Lot of these plugins.

C

B

A

It's interesting because when um dina presented the four keys project just a couple weeks ago, maybe a month or so ago, uh I don't think there was an implementation for jenkins, yet that they'd done.

C

Neither neither was there for like uh techton technically because she did it every she did everything to through big query um and there's not like one. I don't think there's a single, singular thing like a place where cloud events can be uh monitored as such, but uh she so she basically implemented it uh implemented the four keys with tecton.

C

Now here we can think of you know doing that through jenkins, um and you know, instead of taking in uh techno cloud events and bigquery, we are taking in jenkins cloud events in bigquery.

A

Yeah this this could be your contribution. Sweetie.

C

And jeff has worked on get github, auto status.

B

Yeah yeah, that's why I mentioned this way: faithful and different.

C

I'm guessing these meetings might be a little a little too worthy for him.

A

Yeah he'd approved it, but they are early. I think they're crazy early for him like five in the morning or something so we might have to try. At the very least we we may not go to everyone, but we should we should actually I'll put it in the sack. We should make um an aipac friendly meantime because it, it is a shame to be missing out on his expertise.

B

Do you guys think that if we um shift this meeting to like two to three hours, it might work for you.

A

It works for me generally, but two hours well, two hours, four words: um there's there almost every monday, there's a cdf's sake that I think of above and I'm both both involved in either way. They're best practices or events.

D

A

Probably will um might want to sit down too, and but uh one one hour before is early is fine for me and one hour is fine, so one hour or three hours, basically, I'm probably the most flexible. Unless and this is totally possible should we need to we made a meeting time that was that works for aipac and pacific region like where jeff is on the west coast of the us. I think it's just so hard to span.

A

Europe pacific goes to the u.s and then a back to so like india time it's like it's just it's just such a wide spread. It's it's! It's like a teeny bandwidth and it's going to be painful for at least some people, so.

B

Yeah, that's really true, um I think, like the most you know, optimal would be just like little late nights for india evening, slash like um evenings for your maybe kind of early morning for the us, but not 5 a.m. Early, hopefully,.

A

So if if we did propose um one hour later or three hours later, three hours later, it's probably getting really late for free all that.

C

One hour later is fine. Three hours later I've been uh I've, been an old man lately, and I've been sleeping at nine in the at night and having dinner at 6 30 in the evening. So I think one hour later, okay.

A

All right I'll propose one hour later, so I hope they will do that from from next.

C

Week, thank you so much guys.

C

So anything else.

B

I think this meeting, I think we can keep about um the jenkins as a source and what we talked about and then um I'll make a pros and cons, but I think I'm going to continue with a listener implementation and then on the side, also keep looking for other implementations and possibly, if you know there's if there is a better way for a different way to work with, we can actually.

A

That sounds great. That sounds very good.

C

Also, keep in mind the sorry. Sorry also keep in mind just like how you're gonna test this stuff. So, like think.

D

As like, a tester.

C

It should be as testable as.

B

A

I was just going to add, please you ask questions on this like channel any questions you have, especially um for jeff in the run-up to hopefully next week's um meeting, which will be got a slightly better time for him, but because you're looking at this plugin that he worked on you.

C

Know jeff has already worked with listeners. I think you can ask him about. You know which one is better, uh so I feel like we already kind of know which one is more used, so we can kind of go with that, because there is some faith in listeners and in people, uh so we can kind of go with that, but I think it's a good idea to ask chef on the channel uh about like jqs versus listeners.

C

If he's considered something else before apart from jqs, which he think might be a good idea to uh you know dive into.

B

Yeah yeah yeah.

B

Similar implementation.

C

B

Well, well, I don't have anything right now. I mean I have things, but I don't think that the the five-minute time that we have left is going to be sufficient.

C

So so about so, your question with your burning desire to be answered right now, is about jenkins, being monitored itself right, like if jenkins, should be uh sending out events on its state. That's what you want to answer.

B

Well, um yeah sort of it was, I think, that's sort of like the year. The conversations, but it got answered by now by burning this, like not the burning desire, but the burning question that I have, in my mind is um just like jenkins is the thing, because that's also something that we need to be. um You know keeping in mind from creating class. We are building it as a source.

B

And and yeah a little bit of a little bit of also what we.

B

C

Okay: let's do that then.

B

Like the meeting the the cloud series meeting, is it.

C

In it's at nine o'clock.

B

All right, so I don't have anything to add.

A

Neither great work this week, I feel like you, are really getting to grips with all the different ways that this has been approached and the various jing games plug-ins, and we seem to be kind of consolidating around the idea of listeners. But it's really good to do the exploratory work, and I guess in that way I understand what a lot of the certainly more recent plugins are. Why they're choosing what they're choosing.

B

They're definitely going to be searching math as well.

A

Thank you everyone. Thank you very much, thanks for being here and uh yes communicate more on the slack channel. If you, if you have any questions or anything like this, is really for you, so we want to answer all your questions. It's not just possible and if you have something yeah.

B

B

Wait were you saying something.

A

No, that was it that was it.

B

C