Jenkins Cloud Native SIG, 26 Jul 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: ☀️ GSoC CloudEvents plugin mentor meeting 2021.7.19

Description

Discussion on architectural choices for the plugin and a decision to start attending the CDF Event SIG 😊

A

uh So so, as I said like you know, pretty related, I'm just thinking that you know for something to be truly sort of cloud native and also build upon that event-driven systems.

A

How can we or how are we thinking of dealing with maybe transient failures or just like simple network failures and sending requests for jenkins as a source, but also for jenkins, is saying: how can we implement an architecture which can persist events and uh and also create a way for any source which is sending events to jenkins sync, to really have that architecture where events are not being lost and every time there is something which is capturing the events and dealing with whatever action needs to be taken at maybe like an asynchronous time, because I keep going back to um that.

A

One thing you said, and that was really that really resonated in terms of the architecture was loosely coupling um and if we basically like design a system where it's sort of like an rpc call between jenkins as a source and whatever sync and jenkins as a sync and whatever source, and it's very like dependent and it's very tightly coupled with another service being available, especially for jenkins and sync or I mean jenkins- is a source.

A

We really want that other service to be available when we are sending to do something- or maybe when we're talking about jenkins as a sync, we really want the sync to be available whenever another source is sending, and I think its implementation for jenkins is the sink- is way more important than jenkins as a source, because you know someone really might depend on taking an action inside of jenkins whenever a cloud event is triggered in that particular source. So just been thinking about that, um there is actually a plug-in, which is it's.

A

um We pops up like pops up light plug-in uh for jenkins, and it's I'm not really sure. If a publish subscribe pattern might work here, having something like a simple queuing system might work or if we, if we even want to you know, go that route for now, maybe just like to hear what your thoughts and that are. If you have suggestions.

B

I mean I, I always thought of it's interesting, this idea of think and source, because I came into the project when I first thought about it was in terms of pubs up so exactly that, but I think there's probably different ways to think about how how events, how they're kept, how you, how much?

B

uh How can I say this, how much you guarantee the ordering of them in many ways and and then how long you persist them for, and I think that's a whole series of. I think there are a couple of different ways of doing it, but I think publish subscribe is, would be certainly a good way to start.

A

um Yeah yeah, I think those are very important things to consider and also when we are talking about just having a published subscribe pattern for jenkins as they sync it.

A

Basically, I like what in my head, I'm thinking what it might look like is you know someone is creating uh a user who wants to use jackets as a sync is creating a topic inside of inside of that jenkins, that that bus, which is going to be different from the the jenkins as a sync implementation or the jenkins in the same configuration, because I don't really think that putting the bus alongside or like the bus or the or just the topic configuration with the actual plugin.

A

If that really makes sense, it has to go, I mean it should come with the plugin, but just the configuration of creating topics which our sync will subscribe to and the publisher will send messages to how. How do you think that can look.

B

B

I think there's a couple ways to approach it. I literally have this data intensive applications open like let's see, let's see, there's a quick idea that I'm doing this.

B

No, I I think, there's like a number of ways in which you can deal with how you process streams, how you store streams like how streams are transported like how are you directly message, brokers, event logs and then how you process them afterwards, like there's so many different ways to approach this that yeah, it is kind of like a really an open question, because you want to leave it like. Is that going to be something? That's left that we want to leave configurable for an end user? Is that something that we decide?

B

I guess if it's just jenkins, it's something that we do need to decide on, and it should make sense for us the icd pipeline, but is that something that should be decided more at the level of like the event seg like? I really. I really don't know what have you? What have you been reading? Just out of curiosity.

A

um Well, I so all of the the usually the pops up patterns of design for any um event, sort of mediator or just a middleware any any service, whether it's from azure or whether it's even a simple thing like a kafka implementing something like that is. Is that usually needing a topic to be configured um and- and you know we cannot- I mean there should be a way that will be open for the users to mention whatever topic.

A

It's going to be that jenkins, as a sink will subscribe to, but also have have a way of persistence in that bus itself, where all of the events are coming in, so actually the k-native broker. I shared this with um with hub when we talk class in our meeting.

A

Let me see if I can.

A

Maybe send it over in this in like in our channel okay, so creative broker or just canadian eventing in general is also very interesting because they have this live system where all of the events are coming in and then they have triggers which, which is essentially maybe you know like like a topic or something that is open for the user, to configure that what do they want to do with this with this type of event?

A

uh But there is that you know middle middleware, which is sitting in between receiving all of those topics and then based on maybe the filters or based on the topic or just based on here for canadian eventing. They have they have triggers, um which is which they say that it it represents. A desire to subscribe to events from a specific broker, so for them trigger, is something which is sort of like sitting at that level between broker and the actual whatever is going to be working with.

A

That will event so they're filtering based on the they have different brokers, maybe configured with um you, know, different sort of maybe metadata or whatever a broker name a broker type. um I think it's usually broker name yeah. It is broken.

B

So super we can, I ask you a question. This is super interesting with the triggers because it doesn't just go from brokers to subscribers the broker emits triggers and then that's what the subscribers subscribe to.

B

So so, how are these triggers? How are these triggers created? That's.

A

So so what so? It's basically like the trigger is sort of sitting at that broker, and it's um so when we have that trigger. It's basically saying that I want to receive event from this particular broker, which is so there is a default broker and I think they call it default. So there is a default broker and then there's broker that that someone can create- and you only maybe as a publisher, only send events to that broker say: let's call it k native um trigger or no no, let's say it's like a pipeline.

A

Tecton pipeline run broker and I'm sending my events to this broker um and then, as a trigger, I've said that as soon as this broker receives any event I want to. I want to subscribe to to that, so that's actually creating a trigger like subscribing to event from a particular broker, and then you know, as as I have subscribed I want to.

A

I want to maybe like take whatever I'm, not I'm not really sure how they are taking actions with their whole, like broker and trigger pattern of a trigger, actually sort of not being a trigger for an event. But being a trigger for for um that, that subscriber or that.

B

Broker is so so so is the trigger. So there's the broker, which is, I guess, like some sort of message, queue message, brass and then the trigger is the logic. Is that right? It's the logic, is the logic of what you do with the different events or what can be done like it's selling to the subscribers like it has more knowledge, more control. More sense of I don't know, I guess more sense of context in many ways like this thing happen, it's not I'm not just giving information.

B

I'm this contains more data on what you may want to do with that information. Is that is that right, or is it not as knowledgeable of the context as I'm thinking right there yeah.

A

That's what like that's what I uh understand of like trigger is just saying that okay, I do want to when, when an event is coming into a particular broker with a particular name. um I am I I'll send this to a subscriber, but there's there's so there's filtering inside of triggers, so you can. You can like filter with different um triggers, but I wouldn't really say that the the the trigger itself has a lot of information, but it does have information for, like let's say the I'm actually looking at.

A

I also said this.

A

Something um one of them they're they're like filtering and how they're doing it so so they have like the broker and then they have filters inside of that broker, which will create that, like that contact or that knowledge that you're talking about. So you know, even when you have a broker, which is subscribing to a particular which is basically sitting between that publish and subscriber of two different kinds, and like publisher, says that you know I have all of these events, all of which will go to broker a even inside of that broker.

A

You will have um like trigger scan filter based on here. They have attributes, so we can also have attributes of you know. The c type should be so, and so the um c source should be so, and so, if that makes sense,.

B

Yeah, thank you that actually does clarify it a lot. Thank you.

B

I didn't know if that helped. You, though, um but I think this is a really interesting pattern.

A

Yeah, I'm just thinking that if we are creating that tight coupling between you know jenkins as a sink and any any such any sort of publisher, you know whatever that source is it's it's very tightly coupled then the whole idea of communication being asynchronous or um being you know, failure resistant is sort of like the whole idea. Is it it's? This loses the whole thing.

A

So I'm not again, I'm not really sure because they're, not just you know, there's women pops up patterns too there, as you said, there's so many um ways we can implement this, and the only one uh thing that I did look into was the the pumps of light plug-in inside of jenkins, um which is like a puffs up. A lightweight published subscribe pattern for jenkins, but it is still it's pretty interesting. So uh it's like, I was just thinking about.

A

What way can we use and how can the configuration look like because you know even the k native event thing it's using that pops up pattern, but how they're creating that topic and creating filters on it is very interesting.

A

um I feel like this will this is going to probably go beyond just um you know the end of gsoc and also talk uh for for, like creating the best architecture possible for venderman systems and asynchronous abandonment systems, creating all these patterns.

A

um I think they take a lot of thought and reiteration, so I'm actually excited to have this conversation and then like have these conversations in the future. Whatever we might end up developing.

B

Yeah, I totally agree. I think this has got. You know. The gsoc project project is amazing and your work is fantastic, but I think there will definitely be more to work on beyond g-suck and you you will be a great person to do it, and I do think it has all these like really interesting questions on architecture and how you make a good event-driven system that our sort of questions the industry as a whole are is grappling with and continually evolving, and it's right in the heart of that so yeah.

B

It is really interesting conversations to have and there's a lot there. It's very cool.

A

Yeah, in fact, um a lot I think I would say, 60 percent of my um the the g song proposal was just proposing different ideas for um designing this event. Sync, because that more than the event, storage, because event source sort of sounds um pretty straightforward, except you know trying for transient failures, but talking about jenkins is a sink there's just so much that we need to um keep in mind. Well, you know we have the event parsing the metadata or the body parsing and then creating filters and creating a whole architecture.

A

So yeah, I'm actually looking forward to the whole jenkins as a sink and everything.

B

Related to it and in terms of those questions around jenkins, as a sync and broader architectural considerations, are you having enough, I guess, support and but, like I know, you're meeting with above a couple times a week. Are you are you?

B

Is there anything more that you need? Are you going to different? Are you going to the event say? Are you meeting with other people like? I think I think these questions are really interesting. We can certainly try and broaden the net of input. You have. Does that make sense.

A

um Yeah, I um have not really been able to connect with like the right people who would have a lot of um sort of idea on this. Regarding the event architecture, I do think that the the event stick um is the most uh relevant for these type of questions, and I'm hoping that you know it'll be my first time today going to the meeting and hearing from them.

A

But I hope that that can be a place where I can um raise situations around this and also have a communication, um and- and yes, they have like talking with you guys and um it is it it sort of gives an idea to to be sort of steered in the right direction of researching, because right now, it's just like looking at a bunch of different things.

A

um So so I yes and no yes, because uh you know they're, like all of the members in the community- uh are very helpful and they're also like guiding for for this particular sort of architecture, but no because I yeah do not really have like been able to connect with or like. I did not actually ask these questions. A lot before is just starting to come around for jenkins as a sink, so I'm hoping once everyone is on the call and also through the sig event or the event sake.

A

Meetups or meetings can get answers to it and have communications about it.

B

Yeah, I think so and the event say is a good place to start and mauricio is, I believe, in his new his new job he's working on k native, so I'm like we might, I mean I realized, he's new he's settling in and that's always such a you know, there's always so much to do there, but um I think, between kind of pulling on more connections that we all might have and then also the events that I think that we could get.

B

You know you speaking to more people and it would be interesting for you um yeah now I want to go to the event's sake too.

B

I haven't had so many other things to do, but I'm like, I really want to go. um Yeah.

A

Because um yeah well, those were the only um top questions. I had uh burning questions. um There's this I'm pretty sort of set. The first thing, obviously, is this doing or moving um the ui from mobile config to having its own page, and so I think this week is going to be doing that and also just researching different um methodologies of implementing um things, but but in a more async communication kind of things, I have been looking at a lot of them and they all have such different architecture.

A

I get so confused, but um but there is should be, and I'm basically like writing down um all different architectures and what can work and what cannot uh so. Hopefully, it should be good for this week and I am meeting with the bell later and maybe I might able to schedule something with mauritia and talk more with these things, but that was that was it for me today.

B

Okay sounds good, um in which case since you have more meetings in the week, we can give you the next half an hour back here, I'll um stop our recording.