Continuous Delivery Foundation CDEventsCon 2022, 13 Jun 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: CDEvents in Tekton: The path to a more interoperable CI/CD system - Vibhav Bobade, Lifesaver, Inc.

Description

CDEvents in Tekton: The path to a more interoperable CI/CD system - Vibhav Bobade, Lifesaver, Inc.

The CloudEvents format was introduced to standardise event formats in the cloud ecosystem. This has allowed different event brokers to consider a single format for communication.

The need for better interoperability between CI/CD systems is gaining importance, as companies continue to have multiple CI/CD systems in their environment.

To facilitate interoperability between them in an event-driven fashion, the CDEvents community was born which would use CloudEvents as the base format for the CDEvents format as a standard for communication between different CI/CD systems.

One such system is Tekton which uses k8s style resources to create CI/CD pipelines. Tekton already supports CloudEvents and there is ongoing work to support CDEvents. In this talk, the speaker will go over the CloudEvents controller for Tekton, work being done to standardise the CDEvents format for Tekton along with a demo for the same.

For more Continuous Delivery Foundation content, check out our blog: https://cd.foundation/blog/

A

So, thank you all for coming uh this afternoon after lunch. I hope, if you are feeling sleepy, um my name is vibhav. I work at lifesaver inc and at livestream rank. We work at uh figuring out the social impact people have from the transactions that they make today uh uh today, I'm going to talk about uh how we are working in tecton to make techton more in more of an interoperable ci cd system.

A

So we are so techton uh is quite a it's quite a new ci cd system in a way but- and it is based off of kubernetes- and uh we are using cd events in tecton to make it more interoperable and in this talk I'll go over it a little more. So what do we mean by interoperability?

A

So interoperability? Just by the word? It means that um we are making two or three systems work together in tandem. You know without having to do too much work and like in the newer system itself. So in this case uh we are taking tekton as an example and showing how we are making data more interoperable.

A

So, let's, uh let's go to techton and like let's look at what interoperability is first and uh so. Techton started its journey with interoperability with just cloud events and uh just defining cloud events, and it kind of uh andrea here like worked, worked very, very heavily on tecton's cloud event system and.

A

In in in this, there are like two aspects that I want to talk about, specifically where which is tecton has the sorry. uh There are two types of interoperability which are like uh direct interoperability and indirect interoperability, where, uh oh, when I say, direct and draw polity, it means that I'm trying to make these two systems work and they have to work, uh and each system has to define how it works with the other system. Very in very specific ways like if I have to com con, convert a docker compost or container communities deployment.

A

I have to show it like, specifically, which part of the compose files get converted into a community's deployment and same in ci cd system like if I have jenkins in my infrastructure and I'm running tecton pipelines as my new cicd system. How does the jenkins file get converted to techton pipeline uh at that point? So this is a lot of work and with direct or structural interop liberty, you have to usually make a lot of tools which, which are very specific to those two systems, whereas in indirect or semantic interoperability.

A

What we do is we kind of define a layer or from which multiple different systems can uh use and like talk from so in this case we the example is like cloud events and cd event, spec and vocabulary so cloud events kind of like for helps us form a base on top of its cd vents is uh built so cloud events is just a spec which kind of helps understand where the semantics start and see the events uh which I'll be talking about after this helps help tecton define how to do next.

A

uh The next part that is talk to other ci cd systems, about uh what is going on in tecton and then communicate, communicate that to other ci cd systems, okay, so interoperability between ci cd systems. In this case, uh imagine you have jenkins already running on your infrastructure and techton. Is this new big thing or a captain or like spinnaker, like any different cicd system is like new and you want to move to that system or want to use new features from it without actually letting go of what you already have in jenkins?

A

So what do you do? In that case? Some of your teammates would be like okay, we don't want to go to this new system. Everything's worked so well. Why would I do this? um Whereas some other uh teammates will say that? Okay, I really want to use tekton because it has really fast build. It allows me to use kubernetes for my bills and deployments and everything I really want to move to it. So in that case you can't make people move from jenkins to tech on forcefully right.

A

You have to have some kind of midway where people can meet and like still use systems together, while they are like moving between systems, so in in that case, uh uh the first step to this would be, like you know, uh doing some kind of like semantic and property like using like cloud events or something and so cloud events helps us, define the basic event protocol and allows us to understand how we can use events to use and to do inter interoperability.

A

um So the reason we are using events to do interoperability is because they're scalable uh they're fault, tolerant. There are a lot of routing methods which already use a lot of events like amq kafka. They already use events, so it's already. People already know uh how to use them then, but the main problem is that uh people have different formats in which they have defined these events for themselves. So there are different ways: they have uh kind of defined.

A

You know what pipelines look like in tecton versus jenkins versus like argo, and this is where we have to kind of bridge the gap. Now, how do we do that?

A

So this is a very semantic based problem, so if we can get the semantic site over here and instead of building structure and inter interoperability by creating, like you, know, jenkins, detecton converters or like uh these pipeline converters, which which will be really hard to make, uh instead of that, if you define just a semantic to kind of get the conversation going between the uh different ci cd systems, in that case, we we will be able to achieve a lot more because we can also have other ci cd systems come into play and then work work together.

A

So in this case there are when we say that semantics should be shared between cicd systems. uh We can think of different ci cds, uh different semantic buckets like core. We can say scm is like one one of them, then continuous integration and then continuous deployment as the other one um in in core, basically pipelines tasks, jobs.

A

All of these course core concepts that are there in, like all ci cd systems, uh can be considered in core, uh then all the scm things such as like uh git repositories, branches, etc, can be considered in the source code version control and then in continuous integration.

A

Like all the builds tests, artifacts such as the images that are deployed uh can be considered an integration where these things need to run in a loop over and over again, then, for continuous deployment, the environments that I use, the staging dev, etc. Like all these environments that I use the services that are created in this case, uh these come under like continuous deployment, so defining this language is very important so that between cicd systems, we know what we are talking about. The cicd systems are talking between each other.

A

So in this specific uh case, we are going to talk about tecton pipelines and how pipeline is like moving towards being a better ci cd uh through being more interoperable, and uh why are we doing this? So, as I gave an example before so uh at the team I used to work in before we used to use a lot of jenkins and overall, the rest of uh rest of our organization started.

A

Moving to tecton, so moving to techtone directly was really hard for us, so we came up with different ideas and uh there was a technoclient plug-in made in the process where we used a technon client plug-in to just uh trigger, like techno jobs through jenkins, but uh that isn't really scalable, because again, that is the example of structural in structural interoperability.

A

So what you have to do in that case is make jenkins and tecton talk through without without that, without that barrier- and you know uh just be able to kind of communicate through a language instead of an api, so I'll just go ahead and describe in a few lines. What tecton is techton is based is a is a cicd system which is which uses kubernetes as like the main infrastructure, and it is all of its main concepts- are written in crds and the pipeline tasks.

A

They are, they are basically custom resources intact on and each step that runs in a task is a container. Each task runs in a pod, and these steps can run in parallel. They can run in, uh they can run one after the other then and each so. All of these are uh basically crs that are created and techton uses, like kubernetes as the main infrastructure, to like kind of run. All these all these things, okay, so so to turn all of this into like an interoperable ci cd system. What do we have to do now?

A

When we talk about events? We say that every time something happens, there needs to be an event created for uh the other system to know that the event event is happening. So, for example, if I run a pipeline and the pipeline goes from like created to failed or like created to successful, uh the other system needs to know that that is happening.

A

So uh till now, uh we've had a internal cloud event, sync, which basically accepts a sync and then sends all the events that are happening for, like the tecton pipelines to the uh particular source that is taking taking in all the to the sync that is taking all these events and then processes them so so in in that we have support.

A

So this is currently built into the tecton controller and the support for these technolon-based cloud events is there and you can describe these things and they'll, although all the events will just go there and you can process process it. However, you want okay. This is what some of these events look like, and uh you can.

A

You can see that there are task runs which which give the state from like started to running to like, if they're in an intermediate stage, which is like unknown, then if they are successful, fail same thing with pipeline runs and runs.

A

So this is what we this is, what the controller can already do out of the box intact on and uh where are we right now so right now we are moving towards the more uh cd events fashion of doing things, so the cloud events that are there are very the language that is used in these cloud. Events is like very technon specific. You can see like dev.tech.event.taskrun.

A

But uh this too, for this to be understood by other ci cd systems using like cd events. uh There needs to be a more uh common language of understanding with, for which we are going to use cd events and uh that that we can that we are doing by using uh cloud events controller that we have that we are making and the cloud events control already exists in the tecton experimental uh experimental repo, which you can go check out later. uh Basically, the controller.

A

What it does is, it extends the tecton functionality and allows, and then so you allows the users to define different syncs, which they can uh define a sync where they can send their cloud events to and in in this control.

A

They can say that I want this to be in a cd events format, so it can either be a legacy or a cd events format, so legacy format is a format that is already supported in tecton, whereas the cd events format is the new format that that is supposed to uh kind of um make the cd event format as like a normal for different ci cds. So um so right now there is support for runs.

A

Task runs and pipeline runs, and we are working more on getting uh getting like more features inside in, in the tecton controller, to be able to support, support like new results and different like being able to relay these uh events like different controllers. So.

A

As I said like, we are working on building like more stuff into the cd events controller to kind of relay. All these results to like different controllers, like uh some of these uh events can be stored as results, and then they can send they can be sent out to other ci cds and then they can be passed over there and all of that can be taken back and then techton can do like deployments and all based on so uh there's a lot of communication.

A

uh There's a lot of scope for growth over here and uh yeah. Another thing that the work there's work going on is uh these events can be sent to like triggers and they can be passed there and and work can be done in this way.

A

So one or some of the stuff that we are working on next is the cd events or controller should be able to keep up with the cd events format, and uh this format would uh keep on changing. It will keep on evolving as time goes on.

A

So that's one of the main goals of this controller project and replacing the cloud events functionality in the controller with the dedicated controller, which means that uh the already existing functionality of parsing cloud events uh would be completely replaced with this new controller for cloud events, and this will be the one that will send all the uh events from the controller to the uh to the other ci cd systems and such so yeah.

A

So elfie, that's the that's the end uh big shout out to the techtron team and the cd events team. For this talk and uh helping me out with this and uh hit us up on the cdf slack uh for cdm and sig and uh you can come contribute to the controller. This is the link uh go head to techton experimental, the cloud events controller and also check out the cd event song.

A

Thank you. So much.

B

A

Any question from.

B

B

Let's audience or from the virtual audience.

B

Thanks again, thank you.

B

um So we have a short well a break now for coffee. We are a bit of.

A

B

Of schedule, so we start back at 3.

B

B

Yes, we will be back here at 3, 30.

B

B