Jenkins Cloud Native SIG, 29 Jan 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: 2021.1.29 Further discussion on Cloud Events Plugin proposal and Tekton Client Plugin

Description

No description was provided for this meeting.
If this is YOUR meeting, an easy way to fix this is to add a description to your video, wherever mtngs.io found it (probably YouTube).

A

So what we have been discussing, do you all have access to the meeting notes it's on the invite, but I'll stick this in the chat as well, and then what we have been discussing in past meetings is um the cloud a little bit the tekton client plug-in, but also um the cloud events plug-in proposed by vibhav um and there's more on that in the meeting notes. To give you a bit of context, if you are not familiar with those initiatives.

B

B

B

Okay, I guess then it's fine, okay, okay! So hey guys nice to see you! um uh So, let's, let's get started.

A

Again, I will screen.

A

A

We can go over the action items from last week's meeting, so we were going to do some research on cloud events metadata.

A

What that would contain um think about user stories around cloud events, the number of action items we have here actually and the play with the cloud event. Sdk.

A

B

Yeah, uh so um so here, okay, so research needed on what cloud events metadata should contain. So this one uh I saw around a bit and saw how if the events already been collected and uh sent to a place uh through jenkins, so it turns out there is a plugin called statistics, so statistics handler or something so let me just I just have it right here.

B

Oh yeah, it's the statistics. This plugin basically uh could help us get started with like what all stuff we can uh send us cloud events, uh then uh what I was thinking with this was we could probably just uh start with getting these uh events with the start. District gathered already gathers and then uh convert them to cloud events, the plugin. So this is what I was thinking about, and this would be a good start basically for this uh and yeah.

A

B

Is that is for that? Let me just open the dock, so I can just yeah okay, so here yeah and then there is uh so that was that was some of the research. I did then also uh the metadata, so I still have to make the metadata table and like what that looks. Like uh other things, I have been on uh vacation for this last week, not vacation, but uh last two days I was working, but before I had been on vacation.

B

With respect to the cloud events sdk, so I read the code in the examples for the cloud and sdk and seems rather straightforward like how we could actually uh change the jenkins uh uh like job. uh Like start, stop finish like or whatever the events that are there from the start, statistics gather plug-in. We could actually just uh convert them to uh cloud events directly. uh We should we just have to uh see how to extend the other plugin for that and we could kind of start out over there.

B

So this also kind of falls under what could be the um extend of the bootstrap, so the bootstrap uh can contain a basic.

A

B

Cloud ancestry uh like use it of cloud events, http basic module, and then the student can go ahead and see how that works, and then uh they can play around with the statistics gather plugin. We can help them out with uh how to get these, uh get these statistics and then convert them to cloud events, and then what the meter data should look like then. So.

C

B

This is what I was thinking we could do and.

B

So the road uh okay. So what is next so find end points that can be subscribed.

A

Can I can I ask you on what you were just saying: um this is.

B

A

Me and it might be helpful to others um when you talk about taking the events that we can already sort of register and transforming them to cloud events. What what is involved in that process of transformation.

B

um Good question so that would be uh reading through the statistics of the plug-in code and uh the status of other plugin has dsl, which uh people can already use. So it is clue it has the synth as well that you can give- and this sync uh like is given in statistics login and all the events regarding job project creation or anything that have that the user has subscribed to. They just go to that thing, so the tran. So what so?

B

I I don't have a detailed idea about like what goes into uh the transformation exactly, but this would mean that uh the user uh student.

A

B

Extend uh classes from the statistics of the plugin and then work them out in the uh work them out in the uh cloud cloud events plugin itself, so that would be the high level process. But uh I uh what I'll do is I'll try to figure out what is the exact detail process and uh I uh I can update you guys so like on the channel or the next week.

B

Does that answer your question.

A

Yes, it does. That would be awesome as well. Thank you.

B

Thank you uh so so the fine, so the next one we have is uh find endpoints that can be subscribed to so uh this one. I will sing over it a little bit because this this one is more related to consuming cloud events, and uh um I am not sure how that would work exactly because I haven't worked on something like that before so guys.

B

Do you have an idea how we could probably consume cloud events, maybe into jobs or something uh would it be uh so do we consume and then convert the cloud events into jenkins metadata, which can then be used to trigger jobs, probably like or trigger something uh like a generic web trigger, or something like that.

D

Yeah, I'm wondering um I mean this. It's definitely something to like investigate, but I'm like initially I'm kind of thinking is. Would this be kind of just like a web hook handler that we can post cloud events to in a similar way to like the github.

B

D

D

There'd be some level of authentication, I'm assuming on that, like hmac token or something yeah um yeah, some something along those lines. um That might be a good first pass and there are plugins out there already that do something very similar. So.

B

So there's a generic web trigger plugin, uh which also mark had suggested to look at this one. I still have to look at it so.

A

He suggested that.

B

Maybe this could be a good place to start to understand like how the consuming of cloud events would work in jenkins. So we could probably look at. That is an action. Interconnection.

E

And yep plugin is called uh what is the name of that.

B

Yeah generate webhook.

B

B

Okay, so, and uh next one next one we have.

B

Okay, so next one, we have first prototype to understand how cloud events work. So this uh this prototype, I feel that would be part of the uh bootstrap, uh so this bootstrap would uh basically yeah. This bootstrap, as I said, would basically contain like a minimal like cloud events, hdb leader and handler, uh like handler, uh which, which would just like, be based on the uh http cloud events http basic module, uh which we can just create so that so that would be like a good place for the for the student to.

A

B

And I I had a question about this actually so uh cloud events. uh Instead of it being a build stack, would it just be a global plug-in configuration like uh work in the global plug-in configuration it would be like uh the admin should be able to uh enable like certain cloud events for the jenkins, so he would go to plug-in configuration.

B

You would go into cloud events configuration and under that he would just uh mark. You know, okay, I need to see jobs in the cloud events. I need to see uh job steps also, maybe like how they are being executed. They fail or something like, and then there is project. Maybe I need to see what projects are being created so.

A

B

Like uh this, why I thought about this was we could make build steps as well, but I don't, but I think uh global plug-in configuration would make more sense and it would be quite uh should be easier for the user to have, like uh you know, like just have like proper configuration for the jenkins itself.

C

Let's say github is give her build strategies or something about a plug-in.

D

Okay check strategy, or something like that that sends um it sends github sort of checks on every build stage. That happens, and I'm wondering whether there'd be some useful code inside there um to listen to, for. If that's just like you install the plugin and every build that happens, automatically gets.

D

It might be um something similar to listen for.

D

It creates a check for every stage that happens in a pipeline. Basically.

B

Okay, that's the consumer aspect of it: you're watching: okay, okay, okay, okay yeah! That makes sense. So I was actually talking about the producer aspect of it. So to produce like the plugins like so produce, the uh events so probably having a global plug-in configuration would make sense. That is what I was thinking.

B

That makes sense right for the production.

D

I think so yeah.

B

Okay, and- and in so that is producing and consuming uh yours you're saying that we should have. uh We should look at the that plugin you're talking about for this. This plugin can. Can you? Can you remind me what it's called.

D

Give me a few minutes: I've got it here. I've got it locally. Actually, I can't remember the exact.

D

D

D

I think it is just the github checks, one.

B

B

There is a github checks.

E

D

That's the github, auto status, plugin.

B

D

So that's that can be used with a few different things. Actually.

E

Can you put it in a link in the chat, okay,.

D

Yep, certainly, I've got far too many windows open. I know.

B

B

B

B

So this one, this one would be so pretty uh consuming.

B

B

Okay, okay uh I'll actually have to look into this.

B

What does it mean by autocad jenkins plugin to provide automatic status for multiple answers? What.

D

So I think it kind of it kind of sends events to something um based on whenever you, whenever a bill or pipeline hits a or runs a stage. So I think it's used for like reporting into things like yeah influx db and grafana, or something like that. But when I think it's installed with the checks plugin, it sends that to github so that if you've got a pipeline with 10 stages, it will appear as 10 different checks. So you can kind of like follow the status of your pipeline through.

B

We could actually follow the model for this to understand how consuming would work here that.

D

D

How it's obviously listening to the jenkins internals to trigger, as I think, would be the interesting part.

B

B

B

Next one, um let me see yeah, so next one is figure out extend to which bootstrap for the g-stock students, user yeah, so the next, uh the bootstrap would be basically just implementing their cloud. Events, hdb basic uh module or to like a very simple level and in the plug-in and uh like implementing that in the uh global plug-in configuration that that would.

A

B

And actually, this is this is actually related to uh your last question.

A

B

Like how do you see the transformation from the statistics, gatherer plug-in to a cloud events plug-in so probably would need a little bit more homework on this part, uh because that would tell us like uh what the bootstrap would look like, uh because I think we'll have to bootstrap the global plugin configuration and.

B

After we are done with that later on, we have to guide them with how the transformation would look like so to get a better.

B

And then yeah, so the next one is create a draft on technolon, plugin and okay. I haven't shared it, but I have a draft here.

A

Would you like to share.

B

B

I'll share it uh share it in the doc itself.

B

B

There are so there is quickstart. I still have to write implementation examples and probably add a few more links.

A

So uh have you bought I've had to request access, maybe maybe that needs to be shifted. Do you.

B

A

B

Can you can you see it no.

A

B

E

B

Okay, so what's next.

A

Should we take a look at this I'll share if you.

B

A

Have you um you've had a look at the metadata that say tacton is producing, because I think for the cloud events work. You had said. That was an initial first step and I think you've you've probably already done that, but that might be initial first step to outline for students or to have them do produce hack.

B

Yeah so yeah I did that and uh the metadata, so I so I have written down the metadata and it's in a text over here. I didn't put anything on google doc like these two, uh so it's basically would be like.

B

Dot io.jenkins.event.com dot uh uh created and that would be succeeded and all that would be failed. So so it would be something like that, so I still have to write the table itself, so that would be the action item which should be moving. So I couldn't complete that action item properly. I kind of got an idea what it would look like, but uh I still have to write the metadata table down so I'll. Do that I'll I'll? Do that and post in the media? Should I upload should what I'll actually do.

A

B

Create a doc in which I'll just note down everything that is required for the cloud events plug-in the extent of the bootstrap and everything I'll just uh make like good notes about it, instead of having it in text. So like actual text, so I'll do that.

B

And put down I'll put that down as an action.

A

That would be great. That would be really really helpful. um I think, for students and for all of us and then staying on the same.

B

B

A

For the taxon client plug-in proposal, um gareth had mentioned the potential integration between tectonis code and the tactile client plug-in. I I haven't had a chance to look into it um gareth. Do you want to speak more to that or above.

D

I mean yeah, I I was wondering whether I suppose we could use the idea of the dot techton folder um and apply those type of changes. um I did speak to the guys on the drinking sex project, about how they're, manipulating and modifying it within lighthouse, which was very similar sort of idea.

D

Actually, um so I think that kind of thing could work quite nicely. I think there would need to be some processing of resources, but it's fairly minimal um in this case of yeah kicking off the pipeline run with the correct variable set and things like that, but it should be.

A

Is that what you mean by processing of resources.

D

Yeah, so they have a, they have a um a lighthouse folder which contains uh the tasks and the runs, and it's just a list of yammer files to be applied, but some of them, I think they're kind of like load they're, not just applied loaded, manipulated slightly and then applied, um and it's what level does that have to have to happen, um and that's listening, look from looking yeah, so yes also so that the tech timer's code protect yeah, take them as code projects seem to you had to create a a pipeline yaml and then a tasks yeah there were certain file names you needed to create, but, but generally the approach is very, very similar of how it's doing it here.

A

Okay, awesome and the above just so I know, and everyone else so you know um lighthouse is functionally similar to prom. Was that a fair enough way to say it.

D

Yeah, it's it's more of a um yeah tecton native version of pro really it supports tecton as a more first-class citizen, nice.

B

um So how so? How do you see the integration uh going about like uh like we have some technical idea like what that would look like.

D

No not yet not yet.

D

um I suppose it would need I mean it would need to handle whatever the incoming events are to trigger the job. And then it's going to need to clone the repository, presumably on on the master or on the controller and then load any. If that tecton folder or the tecton folder exists, um load any resources from there process them and then apply them into the cluster.

D

And there'll be some configuration like what name space and things like that. Do you want text on builds to run in um because you may want to separate them? Have them run in a separate name, space from where, certainly from where jenkins is running, or where is text on itself installed?.

D

I mean, I suppose you could see longer term that people may want to actually invoke tecton jobs in a different and different clusters, um but certainly not very.

D

B

um I'm I'm still still thinking what.

B

For me, like thinking in java, is like a little tougher than thinking, I'm just thinking what that would work. Like, um probably probably could you uh uh could? We have an action item in which, uh like we understand technically like how that would uh work.

D

Yeah, that's good.

B

Because uh in my in my head, when I'm saying technocrat, but I I I just see the ui and so I'm just having like a hard time understanding like and uh when it comes to technology, it just uh it's, they seem like on like two different planets to me. You know, uh because the client plug-in basically uh would create stuff uh and.

A

Probably allow.

B

To have dsl, which would be used in jenkins fire at some point in time, uh so.

B

I don't know how that would work. So when are you saying you want to integrate with tecton on the code through no probably not.

D

I should probably not integrate so that the from so from my understanding, the tech time as code piece is, is a poc. At the moment, it's like a um someone's pushed it up as a proposal as like. We could do something along those lines and I think that approach.

D

I think that approach would work really well, so um I don't think integrating- and I think you know directly integrating with what the work that has been done for the technology code, but I think doing something very similar, something along those lines, so yeah having a tech, ton, folder and being able to, I suppose, test for the existence of that. If it's there then apply those resources.

C

D

I actually have a poc of that working um from a jenkins pipeline, but the problem is it: it spins up an extra agent to be able to do the applying of things and that's one of the things I was trying to avoid because it's an agent, it's a jvm, it's another pod to do the applying of stuff and then the monitoring of that job is.

D

A

Can I ask a more basic question: what would be the end game for that like what would be the user benefit.

D

Of using tecton in general, with jenkins or.

A

Of of doing this combination like what.

D

um So one of the things that um one of the tech time sports really well is um the tecton catalogues and the sort of reusable tasks um and they've got this templating stuff. Now, that's quite nice, so you can pull in existing tasks from other gate repos and have them versioned and synchronize them, and I think you do the same with pipelines as well. It's almost like a you know a slightly it's a different implement, it's more of a cloud native implementation of pipeline libraries.

A

So that's what the tech change the text on client plugin helps in that space. Yeah the text on.

F

This code, what so tectonic takes on this code, is a proposal to kind of.

D

Yeah, it's it's the kind of inception problem you have with with any of these things right. It's a way. Does it nerf? How does it? How does it kick itself off?

D

um So that's like just that's a proposal of how they would how they could extend the tecton project to do that um and it's actually what lighthouse did it's very, very, very similar to what lighthouse have done to be able to create the pipelines in the first place.

D

um So the proposal, I suppose, was to extend the current text on client plugin to do that.

B

D

Or something similar to that yeah.

A

Okay, thank you that's more clear for me.

B

um Probably uh probably more uh idea about this, I'm still a little bit uh confused uh like I. I still am like trying to piece the puzzle together, trying to understand how the level, but when you say that uh you're trying to uh extend the electron plant again to do something similar as uh the tectonics code, but in this in this scenario, I don't see the dot tecton folder.

B

What I see is, uh I see a dsl in a jenkins file which, which is uh which the user can use because of the techground plan. Plugin I may be the dog tecton folder can be a place where they can keep the resources for the uh jenkins file to consume.

B

But this this is, this is close to it. As I can see it not. I am not sure how the dot I probably need to read more about it, uh see how lighthouse.

B

B

Yeah, so should we should we add an action item for this so to understand these this one or the other better like how this should work out.

A

A

I'll, let you I'll let you.

B

Probably this would be more of a team action item.

A

B

B

Smaller just send one of.

C

A

B

A

The james's and see if they want to discuss the protection, client plug-in and the tactileness code, because that might they might be a good resource also andrew, bear might as well since he's worked on my house.

B

Yeah yeah, oh yeah, yeah. Actually, that would be. That would be good actually to get a bit of light. How does lighthouse exactly work? uh Do you have any idea like I? I don't have like a perfect idea of how technology code works as well, and you were saying that lighthouse works based on the same principles so.

D

Yeah so lighthouse um it was originally written as a kind of replacement for prowl um where it takes in your github or bitbucket or whatever um web hook. It converts it into a like an internal representation. Really it has a series of plugins that can be run against that data.

D

And then originally it used a jenkins example, which was like a jenkins x's own, almost like dsl, for describing pipelines that it then converted into a tecton job.

D

But what they found is that they spent so much time, essentially like version chasing or trying to keep up with the tecton syntax, because it's evolving and it's you know they had to build that into the jenkins x, syntax and it very was very time consuming. So it was felt that it was. It was better to use text on directly. So what they have is a dot lighthouse folder, where you put your tasks and pipelines and things and when a job needs to be triggered, it loads.

D

The text on resources from there and there's a slight bit of manipulation, but then applies them in the cluster.

B

Do we use customized by enhancement that they take those technology they use economic patches or some something like that.

D

I mean, I don't think they use customize at that point, I think they have been using customizing captain things in other other parts of the um product, but I'm not sure if you, I think the lighthouse was designed to kind of be a bit agnostic to all of those things.

B

D

B

Again, probably also yeah then definitely need to read more about it.

D

It was, it was originally kind of designed to be like a high availability web webhook handler for github, so that when your cluster was updating or if anything died at least you would still record webhooks or capture them and store them. They stored it like internally and they can be processed. That was the idea, um and then it would sort of eventually catch up.

B

B

So, okay, so should we move on to the next one?

B

Oh yeah, okay, so the last one was uh would be great if james is for attending to discuss technology.

B

uh That would be great, indeed, yeah.

A

I would when and also andrew if he could speak on lighthouse. I will ask him as well, so I put an action item for myself to do that.

B

Oh okay, hopefully we'll.

A

Get them I know, they've been really busy with um the general well of alpha of edging as x3, but hopefully we'll get them to come in. Maybe.

B

Yeah, I will become one.

A

Could any other questions or or action items anything else to bring up.

E

Do you have anymore.

A

No okay cigar did you have any questions.

G

um um No um because um I don't have even currently even much idea, I'm just trying to figure out. um I just wanna work explore cloud events so yeah later on, maybe, but not currently.

A

Okay, nice- and do you have do you have a good sense of like first steps for for exploring cloud events and how you would move in that direction?.

G

um Currently, what I have a plan is um exploring kubernetes and then um reading about cloud events by own, because um I mean um some steps. That would be also great.

A

I I think um explain: kubernetes is a very good first step um and then you can look at some of the resources we put in the talk which another feedback. You would like to add anything to that.

B

Yeah uh so into that talk uh actually, so if you want to get started with cloud events, so that doesn't actually require kubernetes knowledge, you could actually just uh play around with the cloud events sdk and see just go through the example see what it looks like just play around. I would give like a really good idea how to start cloud events.

B

uh If you want to see the tecton plant plugin, then you can, then you would need kubernetes and act on on, like literal from that knowledge, that'll be nice. If you would want to do it.

G

Okay, so um I will um explore the cloud events and then, when I have some idea what it is actually doing, then I will maybe later on. Then I have a report.

G

A

You awesome so those were going over all our action items from from last week. We have many more for next week, but it's good. I feel like we're, making good progress and understanding the problem space. So that's excellent. um Anything else. Last minute additions, questions, anything.

A

All right, good, good meeting. Thank you so much for being here. Thank you um for working on these projects. It's really interesting and see you next week.

B

See you guys, I'm gonna go around the beach bye.