Continuous Delivery Foundation CD Summit 2022 - Detroit, 4 Nov 2022

Previous Meeting

⏯

youtube image

►

From YouTube: Workflows - Quick, Easy CI with Tekton - Dibyo Mukherjee, Google

Description

For more Continuous Delivery Foundation content, check out our blog: https://cd.foundation/blog/

Workflows - Quick, Easy CI with Tekton - Dibyo Mukherjee, Google

Do you want Tekton's flexibility and power without having to manage lots of YAML? Tekton Workflows is an experimental project that makes it simple to set up your most common CI workflows from a single place. In this talk, we'll describe why we decided to build the Workflow abstraction, how it builds on top of existing Tekton resources, and how it can simplify your Tekton usage for many common CI scenarios. We'll discuss how Workflows automates git based triggering, manages concurrent pipeline runs and provides status notifications while still allowing users to use the full power of Tekton Pipelines!

A

Cool all right, I'll get started, uh welcome everyone. uh My name is W Mukherjee I am a software engineer at Google and I work on tecton, and this talk is about workflows, which is sort of this new experiment that we've been running in tecton for a little bit. So I wanted to like give like a brief overview of like what workflows is and like what kind of problems we are solving with it. Etc things like that: okay, okay, so I think you've probably heard about you know.

A

Tecton mentioned a couple of times today, uh so at at a very high level. Tecton is a cloud native, uh flexible and Powerful like framework for building CI CD systems, and it sort of consists of this like bunch of like different projects that are all sort of like Loosely coupled and they all sort of like come together and you can like Stitch them together to build like really powerful, bespoke CI systems. You know we heard earlier in the day about like how one size does not fit all you know Etc.

A

So, like you know, we have like great examples of like folks, like solarwinds Etc, using like pipelines in a combination with like chains Etc to build like uh really powerful, like Ci systems. Okay, so uh this slide basically contains, like you know, just like the four projects that I have that happen to have their own specialized logos.

A

But there is like definitely more projects too Tech turn like results Etc, uh so you know uh it's a whole pipelines is sort of at the core of tectons, and then you have like triggers or Eventing chains for supply, chain, Hub and catalog for reusable uh and shared tasks right, uh but as like someone who is new to techtown or even as, like you know, members of the tecton community who have been working uh using tecton for a while, like the question really is like how do you get started with all these different things?

A

Like you? Have all these little like little pieces and, like you know it would be nice if you had like a little blueprint or something that could help you out. You know. So uh you know we have had some experience like you know, running tecton ourselves as like part of the tecton community. We use tecton for our own CI CD and we've also spent some time uh talking to other.

A

You know users of tecton and see how they've been using uh it, and you know, while people do like very interesting things on their own they're, you know like some similar patterns, that sort of like uh emerge out of it. um uh First of all, like you know, a lot of tecton users tend to be like Ci Builders.

A

These are you know the software Engineers or developers who are responsible for building out and maintaining your CI system and what they end up doing is like they will, you know, run tecton as a service and they might provide. You know, build some sort of an abstraction on top or over tucked onto like, provide it to the end users so that they can only use like you know, uh like the guided path sort of for tecton.

A

um Another sort of pattern is like common modules and common module sounds very generic, but it is like very generic. You know people build very different things. Some people build like specialized tasks, others build like integration with like specialized third-party systems. A really common one is like Integrations, with like uh GitHub or like other source control systems for notifications and triggering Source control really is you know, like very key uh to uh you, know any CI system.

A

That's where, like you know, specifically like we notice a lot of GitHub, GitHub apps and, like checks, API usage Etc. So that's that's. I wanted to specially highlight that, and obviously, like tecton being a crd, you know residing in sort of the kubernetes ecosystem. There is like a lot of yaml that, like usually Engineers tend to manage um the little diagram on the right is actually uh from tecton's own dog fooding, like tecton, using tecton for its own, like Ci CD. uh This came from Andrea like one of our contributors from like a talk.

A

We did like a couple years back so if that looks a little complicated, this is actually not the full picture. There's like more bits to it. So there's, like you, know, lots of little pieces that you have to like string together right.

A

um So um our goal with like workflows, the experiment is to like really have like sort of a blueprint for all uh these projects in tecton, where you can, like you know, if you're getting started, we sort of provide like a little template for you to like use tecton a for like most of your CI use cases. You know it's it's going to be impossible to like build something that you know is uh that works for everyone all the time. So we sort of hope that workflows will you know uh work for some use.

A

Cases like you know, get you like most of the way there like where I'm thinking of people like us, the Techtron Community for our own, like open source CI needs and for others like if you're building out in in, like a more like controlled environment, where you're more you know, restrictions compliance whatnot, you can sort of use workflows as sort of like your template or your SDK to like build things up. uh So what does this actually contain?

A

So, uh first of all, you know there's like more yaml, but this is like sort of like uh we are abstracting like providing a place where you can declare all of your like Ci end-to-end, not just like the pipelines, part of it, but all of it.

A

um The second sort of thing we provide is uh that is Source control integration that is like well built in, like we have protect on Primitives that can like do triggering and that can verify payloads from like GitHub, and that can do notifications, but like this is sort of like a all-in-one thing that, like you know, sets up triggering for you uh provides notifications at the end of your runs all by default.

A

uh There's other helpful features that are sort of like built in so, like you know, things like uh pipeline support directly from git uh concurrency support, which is like something we have noticed. A bunch of people bring out so build out. uh So things like uh you know. If, if a new commit is pushed to the same uh pull request, you know cancel the previous run or like. If there's like multiple pipeline runs of the same pipeline, you know queue them up.

A

You know things like that: uh persistent log support which and we've been you know like if your pods get deleted from the cluster, you still want to like retrieve the logs for later analysis, Etc stuff, like that uh simpler filter, so you don't have to you know, write Cel for doing like simple filtering stuff um yeah. So that's that's like workflows in a nutshell, uh the project. You know we started out as like a proposal in 2021 and since then, we've been like sort of working with like the larger tecton Community. uh You know.

A

We've been working pretty closely with uh openshift pipelences code, which sort of provides, like you know, like opinionated CI. On top, like we've been like human on team, we've been like providing us with great feedback, uh and then we sort of like shifted to, like you know, construction mode. So we've been like building out this experimental project.

A

um It's still pretty early days. You know, like the yaml I showed in the earlier slide, is like what we are tentatively building out. It's like our first version, but there's probably like other features, Etc that can come later uh so yeah uh we have. We have a little, you know working group. We have a slack Etc where you can come join in uh because the project is so like early. uh You know any kind of feedback that you have as like. You know.

A

Ci practitioners from like you know, building complex the ICD systems would be super helpful and we'd, be you know very. You know happy to get more feedback for features. We should prioritize like things we should build out. uh You know so with the end goal. You know like you, you can still get started and build CI on tecton pretty quickly and easily, while still preserving, like you know the power and flexibility that tecton is really known for uh so that's it I think that was about five minutes.

A

I, don't know if you have time for questions, but if not okay I see two hands, I, don't know. If do we have time for questions all right, perfect.

A

A

It yeah uh so check down is very like API, driven and so like the core pipelines project is really just. You know, like uh pipelines, that you can execute in kubernetes, essentially well or other places like, but like it's like. You know, complex pipelines that you can execute, and then we have other projects like sort of a more of a plug-in model. Like you know you can like uh mix and match whatever triggering solution you want to like fit in.

A

We provide a triggers uh product, but like there's, like other things, you can mix and match like trigger your pipelines, same thing for collecting results uh like you know, supply chain, you know, like chains and other things just use the public Tech on apis to observe and sign thing exactly. You can exactly yeah and you know like it yeah anything you want. You can like basically like a piece of it. Plugins yeah, and there was another question.

A

Because you mentioned that, because at the early stage of playing this, it might be a politically.

A

uh Why aren't you allowing yourself before yeah, yeah, yeah, I, think I? Think that's a good question, so uh I think uh Argo CD, so it basically comes down to like what your use cases is so like a workflows, Argo workflows tends to be like a little bit more generic. While you know rollouts and Argo CD are very like opinionated versions, uh and our goal with like tecton workflows, is to like uh basically provide like a paved path for like building on top of tecton right. So we have actually taken a look at like flux.

A

You know Source crds Etc, so those are all like ideas we consider we may you know one of our experiments that we pocs that we did actually use like flux for sort of like building some of this out. So like yeah, we're definitely aware of everything that's going on in the system. We look at it as like a place to get like lessons from where we build it out, use it where we can so yeah. That's that's sort of the and then.

A

uh Great question: uh it's too early to say, but I I would be delighted to like you know, get that feature request. Once we have this up and running and yeah definitely a visualization layer is like would be amazing, yeah.

A

Anything else all right. Thank you. So much.