Jenkins Online Meetup, 14 May 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Using the Tekton Client Plugin for Jenkins

Description

Recording of the online Meetup on May 14, 2021. Presenters: Vibhav Bobade, James Strachan, Gareth Evans.

Kubernetes has become the staple for cloud native infrastructure today. With the rise of the microservice architecture and focus on container based serverless solutions, CI/CD solutions also have to adhere to what works best on Kubernetes.

Tekton is a Pipeline engine which checks all the boxes for a performant Cloud Native CI/CD solution. It provides users the ability to create composable pipelines which run on serverless workloads in Kubernetes. And now it is possible to use Tekton itself via Jenkins. With the Tekton Client Plugin, the user would be able to manage Tekton resources from their Jenkins instance and run Tekton Pipelines in the cloud. In this meetup we will discuss and demo how one can configure the Tekton Client Plugin and use it to deploy to Kubernetes in an efficient and Cloud Native fashion.

References:
* Meetup page: https://www.meetup.com/Jenkins-online-meetup/events/278067201/
* Slides: https://docs.google.com/presentation/d/1voA1RR2U74GG0rzd4BGruslWgUmIZWR69n8mDqo2POk/edit?usp=sharing
* Plugin repo: https://github.com/jenkinsci/tekton-client-plugin
* Tekton: https://tekton.dev/
* Blogpost by James Strachan: https://cd.foundation/blog/2020/11/05/bridging-the-gap-with-tekton-client-plugin-for-jenkins/

Join the Cloud Native SIG channels to meet presenters and to ask any questions! https://www.jenkins.io/sigs/cloud-native/

A

Hello welcome everyone welcome to the jenkins online meetup. um Today we will be talking about uh using the tone clan plugin with jenkins and using tipton from jenkins our presenters today, ivy power for james and garrett. Thanks a lot for your time and for joining the call- and it's like this.

A

Okay, so today we have three meetup hosts.

A

We will be doing all the logistics and helping with quinney and the discussion. Let's go ahead.

A

Next slide: okay, so yeah, just quick summary about jenkins online meetup, so it's amitabh organized by jinx contributors for users, so we mostly focus on case studies. Bar stories on presentations, live demos and live discussion, so you can see from the slide that we don't care too much about slides, but we really care about the discussion and we, you know what you put to participate in the chat and we will have connie there.

A

If you want to know more about jenkins online meetup, there is a meetup page, which is basically our lending, and there is also online metaphor description on the events page with more information. Next slide, please, okay, just quick heads up there will be a few other events soon, so the upcoming webinar um is on may 27th about kubernetes creator for jenkins this time it will be in emea and in a america's time zone and and, as you may have noticed, we are recovering the series of jenkins and kubernetes meetups, so stay tuned for more announcements.

A

uh Also, there is uh tips, fear and jenkins meetup. Coming soon uh in june, we will have cdcon with a lot of jenkins content. This is a free to participate online conference, so please register and yeah later we'll send links.

A

Then, on june 25th we'll have jinx contributor summit. One of the topics today will be jenkins on cloud platforms, including kubernetes, and, and we plan to have discussions about tickton and jenkins interoperability there. So, if you're interested in this topic, please join and in september you will have devops world again with strong community agenda. Over the past two weeks, we've been doing junkies online meetups, uh describing community agenda and call for papers. So please take a look and it's still possible to submit your talks. There next slide so and yeah.

A

Since we are talking about jenkins on kubernetes, uh we are looking for speakers, so any topics related to jenkins and kubernetes are welcome. So we have case studies, whether it's automation, plugins for kubernetes. We have thousands of them in jenkins and you might be working on something else. Also any possibility systems like jenkins hand, charts operators.

A

Basically anything is interesting as well as integrations with kubernetes tools like let's say, open policy agent parameters and whatever else you have in cloud native ecosystem. So that's the quick introduction. If you want to speak there is a link online. Meetup speaking, I will post a link to slides uh just uh once I finish and we invite any kinds of presentations so kimadri kara. Would you like to add something.

B

So yeah, hello, hello, everyone, so this uh jenkins online community is really interesting community and I also joined recently so like request. If everyone is interested to join the community, please do so. There are a lot of things to work and a lot of things to learn.

C

B

As well so- and it will be fun, learning and a lot of things from the community. So if anyone.

C

B

Please do contact that's friend from my side.

D

Hi all um I'll just say that we also have a cloud native sig, which runs every friday, and I've put the link in the chat it's on the jenkins io site. Among our sigs. We discuss a lot of cloud native initiatives. You can bring your own ideas. We've discussed the text on client plugin there a lot as well as other initiatives that are happening in the cloud native jenkins space. So please do join us.

D

A

And next slide, please so: oh there should have been a kuni slide, which I apparently yeah. It will be just next slide. Sorry, okay, uh so yeah just to quickly summarize how we do this meetup, so speakers will do the presentation and then we will have zoom kuni. So at any time during the meetup, please feel free to ask the question and our meetup host will either process this question offline or ask speakers when there is an opportunity.

A

After the meet up, we will have open discussion. So basically, once we finish with all questions, we will stop the recording and grant everyone voice permissions so that you can ask any question of the record uh related to tikton related to jenkins. Maybe something unrelated uh your work, it's basically our government of after party and of course we invite all speakers to stay as well after the meet up. As kara said, we have um cloud native speak.

A

Please use the link shape in the chat and also there is cloud native editor where you can ask any questions relate to this presentation offline and the results. Interrupt ability seek in the continuous delivery foundation where both jenkins and tekton projects are represented, and this is also a good channel to ask questions. I will add the links during the presentation.

A

So that's it again. Zoom connect for questions and we will have a lot of opportunities for discussion after the webinar and yes, sir information cup slides. If you could return one slide back and yeah, we just want to welcome uh gareth james and vip half thanks a lot again for joining the meetup and for working on the tekton client plugin. It's a great addition to the jenkins ecosystem for everyone who wants to use jenkins in cloud-native environments, and we are looking forward to your presentation.

C

Okay, so uh thank you, everyone for joining today's uh online meetup on using the techcon client plugin. um My name is, and I work at red hat. uh I work with a red hat, given jenkins images and openshift specific plugins in red hat, and I also work on techno on the side so I'll. Let you continue uh change and gather the introductions.

E

Oh, shall I go next, uh I'm james strachan. I used to work at red hat a few years ago and now I work at clubby's and I work mostly on drinking jackson, tecton, but a little bit of jenkins as well.

F

And I'm gareth I've never worked at red hat and I used to work at cloudbees, I'm currently in between roles but yeah. I work at work cloud based on jenkins, x and.

C

Jenkins, let's uh move on to the agenda.

C

uh So today we will be uh going over a few things related to the tecton client plugin. First, we'll understand what tecton itself is get a little bit of an introduction, uh get acquainted with some facts about tecton. Then we'll see uh why we've decided to make this plug-in itself and see the differences between jenkins tecton and some of the paradigms and some problems because of which which which we are trying to solve.

C

Then we look at how to install uh tecton and jenkins so that we can use it with the tecton client plugin, and then we look at a demo which gareth will give for the techcrun transfer again and then, after that, uh we will see what is next for the plugin excel.

C

So yeah, let's start with understanding, tecton and getting to know tecton. So uh tecton is a ci cd tool for kubernetes and it is used to create declarative pipelines with uh crds. Each resource intact on is actually a custom resource definition and kubernetes. So it is, uh it is built using the building blocks that communities itself provides to create something uh likely the custom controller and uh consulting its kubernetes native.

C

It's built with everything that kubernetes provides and uh it's built to be able to provide serverless pipelines, and this is done through using containers uh as individual uh steps and by in tasks and we'll talk about more about that later and it's. uh It also gives a powerful user interface. It has its own command line tool dashboard and you can.

C

You can basically use anything that you can use in kubernetes with technolon using our including our back, and uh there are a lot of integration which you can use, such as custom tasks, which direct with which you can directly use your own other custom resource definitions, to watch them and do all the stuff, but that's a different story. Let's, uh let's move on to uh some of the basic tecton concepts we will be using today and which we'll need to know for using the plugin.

C

So so, uh though, where the most basic element in a pipeline in tecton is a step and a step is basically uh an ex execution that happens in a container, so a step is, uh can be a parallel to the step, can be a container parallel to the task, can be a port and a pipeline basically or is an orchestration of those tasks.

C

So when we define a task, we are basically defined multiple steps which run linearly and each of those steps run in containers of their own, and when we create a task, uh it is basically a template with which we can, uh with which we can use uh to build a task run. A task run is nothing but a port, and when we, when a task run, is defined and all the steps in the task runs are running. These steps are actually running in containers of themselves in the task and board.

C

So that's how the parallels are drawn uh from tecton to kubernetes and when we create a pipeline run a pipeline run, which is a graph of tasks a pipeline run, which is a graph of tasks. It is basically orchestrating all these tasks which are already created and seeing the order of execution and that way.

C

So these are a few concepts which we'll be using today. Pipeline resources was used for inputs and outputs to tasks and pipelines, but now it's deprecated. um Let's, let's move on to a little diagram to see what the execution actually looks like.

C

So when we create a pipeline run, we it is created based off of a pipeline which is already defined to which we can give inputs and outputs and such and each task in a pipeline is uh each task in a pipeline uh runs as a task run after the pipeline run is created and each task run runs uh is nothing but a port, which is that and each step runs as a container.

C

So when you have a task run, it directly draws a parallel to a port in which the steps are running in container stem cells, which can be shared uh which use the shared environment of the or which, whatever we are sharing. In that case, we can share workspaces and everything, and uh you can also see to it that multiple task runs, are sharing or sharing an environment by uh giving them a persistent volume which they can share to to kind of manage resources between themselves and do certain things which need a shared uh context.

C

So this is this is what it looks like when you're running a tekton pipeline, and we- and we would be using this- these concepts to uh to help us run pipelines and kubernetes and tecton helps with that and what the techron client plug-in does. Is it helps you use spectron for this stuff and users don't have to learn uh how to use most of communities. All they need to know is how how to define their pipelines and how simple kubernetes stock works, and they basically then.

C

Run their pipelines on kubernetes after that. So let's move on to like what it looks like uh what it looks like right now to use a tecton, pipe or tecton plug-in versus what it used to look like running jenkins pipelines with concrete james stratton uh would take over.

E

Thank you thank you, so um you might be looking at techton and looking at jenkins and thinking like. What's the point, why is this different? It's the same, but different from thirty thousand feet. It's conceptually the same. It's a way of running pipelines and kubernetes.

E

um It's one of those things that the devil's all in the details that the implementations are completely different, but from 30 000 feet. You can write a pipeline in jenkins and you can write a pipe on the tecton and jenkins can trigger the tecton pipeline so from 3000 feet. It's the same.

E

Where things differ is how it all works under the covers.

E

Now, as you probably know, if you use is quite a bit when you use jenkins and you run lots of chickens pipelines, all of the pipelines run in a single process, the jenkins controller, so the jacob's controller is where all the pipelines run and the drinking controller then orchestrates all of the steps that run in all the pipelines using remoting and chatting us.

E

So, if, even if you're, using jenkins with the kubernetes agent and the kubernetes plug-in you're, spinning up pods on kubernetes, which then chat to the jenkins controller and the juggies controller, is talking all the time about run step. One now run step. Two now run step three, so it's very chatty and the checking control is the single point of failure.

E

There's all sorts of various other issues that jenkins pipeline typically has lots of jenkins plug-ins to be able to implement itself. So you need to align your jenkins file and the agent and the controller. So all the plugins are of the right version and set up correctly so that your pipeline can run with tankton. A tekton pipeline is completely standalone, so each step of each task is completely arbitrary. You can use any image anywhere on the internet for any step, and those images are only used by that one pipeline.

E

So you could have a hundred pipelines running in parallel or with completely different plugins, to use the jinger's term and there's no issue. There's no version conflicts, there's no uh issues about plug-in versions, not matching other plug-in versions. So it's really easy to put all of your versioning in the tekton side of the house and have a very simple controller that doesn't need to change very much so in terms of making it easy to become more cloud native.

E

The more you can make your pipelines completely stand alone, the easier it is to operate your pipelines over time.

E

There's another big difference in the whole networking, so a traditional jenkins pipeline every, if you're using say the kubernetes plug-in or the kubernetes agents. You need to install the jnlp agent into every pod and then there's this remoting chattiness that happens between the controller and the agent to tell it to run each step in turn as an end user.

E

You then have to write, say a pod gmail to define all of your container images, and then your jenkins file has to refer to step containers in your podium, so you're constantly working on two separate files, remembering which step runs in which container name and what was the image again and where is everything installed we take on each step is just completely standalone: it's basically an image and a command. So it's really really easy to define multiple steps, keep them all uh related to each other and separate in a separate file.

E

Let's do the next step. um The next slide. Sorry, so generally think of tekton as similar to jenkins pipelines, but it's just another way of doing it by trying to reuse the kubernetes platform more.

E

So when, uh with a traditional jenkins pipeline, the jenkins controller is starting uh invoking each step, checking what each step does and so on and so forth. When it comes to tecton. Basically, the kubernetes orchestrator is running each pipeline. The tecton controller initializes turns each pipeline into a pod. Once a pod is created, you're just using the normal kubernetes orchestrator.

E

So if a pod dies, kubernetes will automatically restart it, so your pipelines restart themselves magically. If a machine the pipeline is running on dies, it's started automatically on another machine, for you, that's reusing, essentially the kubernetes orchestrator, so your pipelines are serverless in that there's, no long-running controller, that's orchestrating it just kubernetes orchestrates it! So there's no single point of failure per se for your cluster, another big difference and again it's implementation detail from 3000 feet.

E

You don't notice is um the traditional approach with jenkins is tends to put put everything in one big container image, but all the tools you possibly need in one big image with checked on. We tend to avoid kitchen sink images and we just tend to each step we'll just use one particular image like you'll use, the upstream maven image on the upstream node image or the upstream cube ctl image and you'll tie them all together into multiple steps.

E

There are similarities between the two in that there's a shared file system in the jenkins pipeline and a tecton pipeline, um so they are conceptually similar, but in terms of getting the most up, kubernetes they're they're hugely different. Now it's worth remembering by the way, if you're, using jenkins, already and you're using kubernetes- and you may be using the kubernetes plug-in or using kubernetes agents, tecton doesn't take anything away right.

E

We're not saying get rid of what you have to do, but we're saying consider using tekton as a way of improving improving to make your pipelines more self-contained. So each pipeline is truly self-sufficient and is not coupled with anything else. It also makes your pipelines more reliable with jenkins, it's quite easy to write a bad pipeline that takes down your entire controller, which then breaks all of your pipelines with techton. You could write the worst pipeline you could imagine, and if that pipeline dies, that pipeline dies, but it doesn't affect any of the other pipelines.

E

So this site kind of serverless, modeling tecton, does make it easier to develop pipelines that are independent of each other and make it easier to scale all of your pipelines without worrying about how many jenkins controllers you have and so forth.

E

E

Please, oh in fact, let's go over to you gareth for how to install it. Shall we.

F

Yeah, so if this is a um it's a slightly more involved installation than a standard, um a plug-in to get the tecton client plug-in working so we've you, you obviously need access to a kubernetes cluster to get tecton installed and have that running you don't necessarily need to be running um jenkins itself on uh on the same kubernetes cluster. It could be a different cluster, it could even be running on a on a vm somewhere. It doesn't have to be the same thing, but it's just it gives you more options.

F

So what what our recommendation is that we use um we install jenkins and tektor into different namespaces. You can use just use the standard jenkins, helm, charts to install jenkins, keep it nice and simple.

F

The helm chart has the ability to specify additional plugins when you want to install to get automatically downloaded and configured so and and with all of its dependencies that they all need. So you can configure that section um and then, by putting tekton into a separate namespace. Now, when you run the jenkins helm, chart jenkins runs as a drinking service account. So what you'll need to do is give permission for jenkins to be able to access the text and take on resources in the other namespace.

F

If you go on to the um I don't have a link here but onto the um tecton client plugin uh github page. There is a doc with some installation instructions which gives you the role binding, um that you will need to create to allow that access. um But it's pretty straightforward and it you know you can apply that using helm or just a straight cube. Ctl apply. If you need to um some of the demos you can or some of that you want to just try it out if you're running it locally and you've installed.

F

Jenkins manually you'll probably be running as cluster admin, which, in which case you don't really need to give any questions, because you have all the permission you.

F

F

So, just like, I said yep, it's a standard jenkins, ci helm chart. um So this is this is kind of describing the environment that I'm going to do a quick demo on now is anyone james or for bob? Do anyone want to add anything or if we've got any questions to answer before we go into the.

E

Demo, uh we've got one question, I'm just answering it on the chat which was pipeliners code. So how about we do the demo and then we'll keep going with questions at the end.

A

Okay, yeah, that's the best approach, because I believe this question will be answered during the presentation and if anyone has more questions, please don't hesitate to ask them. I will make sure to process all of them. We have plenty of time today for that cool. So I'm.

F

Just gonna stop sharing and share my desktop instead.

F

Just get rid of the slides.

F

F

So I'm just going to show you this, so is that coming through? Okay, somebody give me a thumbs up: yeah cool, okay, great. So this is. uh This is a local test. Server that I have running. um It is using a docker desktop for mac, with its inbuilt sort of kubernetes support. So it's not kind, but it kind of is it's like a local kubernetes cluster uh with docker nodes.

F

um What do they have to do to it to get it working? Well, the only thing you really have to do is install an ingress controller. um That's just the point to remember, because by default everything is running on localhost, but you can't access it. So, just just as a point to note there. um So what I have I have I have I'm gonna show you on some namespaces here.

F

Let's get this in the middle and make it nice and clear.

F

So I have jenkins installed inside the jenkins, namespace and tekton installed inside the tekton pipeline namespace, and you can see the ingress, which is just an nginx ingress controller running there, um it's pretty straightforward. uh What we do is we tend to use a custom image for jenkins to download plugins automatically and build them in, so that we're not downloading on restart. It tends to reduce pod restart time and keep it consistent with the versions that we know are there.

F

um So uh if we go and have a look at jenkins, just see what it looks like switch to kids namespace, you can see here we have we've got a pod running, it's been running a little while it seems to be going. Okay, we've got some agents and it's all configured in a stateful set, which is their new pound chart version three version way of doing it.

F

So one thing to one thing to note is that my because this is running locally and running on a local host, I don't have any web hooks configured from github to talk to this cluster, so I'm going to have to invoke some jobs manually.

F

So what I want to show you? I have a repository here.

F

Go back into my main branch, so I have a repository here where this is my sort of test repo. It is available on github. I can put the links up or you'll be able to see it.

F

Inside the repo I've created a tecton folder with a hello world yaml, and that is a simple there's, a pipeline run which has embedded pipelines and tasks inside it. So if I open this up, you can see I'm generating a name when it comes through. I have configured a workspace. A workspace is, as james was saying earlier, a shared volume that can be used between tasks and it's kind of passed between them.

F

uh You do there are some things that you need to be aware of, um but that they sort of take on uh specific things in the order that tasks run, to make sure that it gets correctly passed and between them, because it will try to sort of optimize the running of the tasks a little bit and then in here I have a pipeline spec. Now I've defined a number of parameters.

F

um Three of these. Now these these values are actually dynamically set by the tecton client plugin. So it will uh clone the repo it it sort of has a look at it. It knows um whether you're building a branch or a pr and it passes that information onto techtom so that you get a correct. You correctly build a sort of a pr or the main branch or, however it is you have configured um so I am this next task is using the get clone task straight from the tech on catalog that I've.

F

So I've installed that task manually into the namespace, and then I can reuse that test by referring to it for the test graph, and I pass in the repo url and the sha that I want to clone, and it will clone that application clone clone the repo for me at a particular point, um either on the branch or on the pull request that I've submitted the next task that I'm running, I want to build the application.

F

I've specified that it wants to run after the fetch repo command and uses the same workspace just to make sure that I have actually got the source there um and I am using a golang image now. This is the one of the really nice things about tekton is I don't have to create a um an image based off? I don't know ubuntu and have everything installed onto it. So all of my build tools and the right version of golang and all of that stuff.

F

I can just use the the proper upstream images that are official and supported. I know exactly what's in them, you know I use golan 115 and I'm I'm basically calling make um it. I will say this: this application is very basic. It just has one file in it, so it doesn't really do much, um but just I just this is just to highlight how it kind of how it works together and then I'm doing a docker build.

F

So I run after I want to build my docker image after I built the application, I'm using the canico task. So kanako is a sort of an in cluster um docker builder. So it means that you don't have to mount the docker agent, the docker, socket, sorry um and you. It runs sort of natively inside a kubernetes cluster um and and yeah. It's a lot more secure than trying to use a docker demon.

F

And just because I haven't got uh the credential to push to my git route to my uh docker hub installed, yet I've just put no push as an extra arg just so I'm building the image, but I'm not doing anything cool. So that's what my pipeline looks like.

F

So I'm going to move this. Let's move that out the way for a second just so we can see it working.

F

I am going to switch to the tecton client.

F

F

I'm just gonna watch the pods that are running in there just so you can see them start up.

F

And hopefully this text is big enough for you to see. If it's not just please give me a shout.

F

And then on this window here I'm gonna watch the pipeline runs, so you can use prs as a shorthand for that there should be nothing there. Yet.

F

So back to my jenkins instance,.

F

Discover my repo onto the main branch and I'm just going to click build now and I'm going to have a look at the logs.

F

So this is a standard pipeline, nice and simple I'll I'll. Show you I'll show you what the pipeline actually looks like to trigger this, but it it could be a freestyle job as well as a pipeline or a multi branch, or this is just standard things to kick it off. You can see the pods have started. The pipelines by runs, have created they're running in a running state. um It has created three pods.

F

It's actually created four pods, because tecton has an affinity assistant to help it help things run on the same nodes, um but we have three different pods one for one to build the application one to dockable and it's it's going through. Now you can see on the on the left hand, side the logs have been streamed nicely, and that is pretty much it.

F

So you can see that that pipeline was was very fast in terms of pod startup time there was no extra jvms or agents that needed to be started to to run that it was very quick. um Obviously, if it's the first time you're running a pipeline, it probably will need to download those docker images. It can take a bit longer, but uh um that's once that, once they're on the node, they're cached and they're nice and quick- and you can see here, you know we have some nice reuse of the components there cool.

F

So I was just going to quickly show.

F

Let's just make this bigger, what the actual pipeline look like for that.

F

So I'm using a jenkins pipeline standard jenkins pipeline, um I'm saying you can run on any agent. I I need to put the checkout sem um step in there, because otherwise I don't get all of the environment variables set. So if you use, if you configure this as a pipeline, you do, if you configure it as a multi-branch pipeline, you don't just something to be aware of.

F

So you need to put that step in there make sure it sets that all the right environment variables that the tecton client plugin can then interrogate and set uh and pass on to techtoni's parameters um and then the the interesting line there is the tecton crate raw, so it it creates a raw resource based off that uh hello, world yaml. It's it's a file. There are other types that it supports.

F

You can inline yaml here if you want, um or you can specify a remote url if you want to download from something straight from the catalog or somewhere else, and I'm specifying a techton namespace here to do this.

F

I've got some. I've got a couple of pr's that I can show on local branches. I'll just show the code, though I think.

F

So this is, this is quite nice, so.

F

What one of the things that techton is really good at is it's kind of like it allows you to build up or compose your own pipelines from multiple tasks that already exist, so somebody could create a a task that is installed into your cluster and that that task can be reused by multiple pipelines and like that's, how that's the sort of basis of the tecton catalog.

F

So one of the quite nice things we have. So we have a I'm just going to show this small piece here. So this is using the jx release version from jx, which is uh which is a plug-in to interrogate git and calculate what the next version for a release should be based on the sort of conventional commit history, and this is a way that you could create a task and and make it sort of share it within a cluster.

F

um If you wanted to um so it creates a next version as a result of this task that can then be used further on in the pipeline.

F

And writing writing to the file text on results next version. That's that's the techno way of sharing almost a sharing state really between steps tasks and how I'm using that here is that when I want to build my docker application now I can refer to the version here.

F

I can pull it straight.

F

Out and the other one I'm going to go to.

F

Is so this is a example of a more complex jenkins file that uses kind of some of the standard pipeline features that you can do so, if I want to, if I wanted to run a different pipeline on on a change request, as as to my main branch on my release branch, you can use standard semantics in here to tell it what to run.

F

So that is pretty much what I have as a demo.

F

I'll stop sharing and switch back to. Does anyone got any questions about the demo whilst or someone here.

A

Yeah there were some questions which uh seem to be related to the demo, so the first question uh was asked: if you want to have a pipeliners code method, how should we get a formal implementation? I believe we answered this question during the presentation. If not, please follow up um so the next question is: should we, how should we decide the choice of tools depends of who are we immigrating from.

A

I guess it's rather question whether we use sticktone or whether we use classification, kids pipeline or maybe contagious under the hood. So what do you think about that?.

E

That's a great question: I think.

E

One of the main things to think about is what jenkins plug-ins are you using like if you specifically want to use a particular jenkins plugin, that's going to dictate how much jenkins file you use versus how much tecton you use the sweet spot. I've checked on these, I just want to run. You know a making go or node, and I want to run a bunch of tools together and then, when that's finished, I might want to do something in jenkins with the jenkins file.

E

So getting that split between the jenkins file and tech ton is really are you using a jenkins plugin a lot if you're, literally just like a little pipelines, you're just doing like a maven, stuff or nerd, stuff or girl, stuff or whatever that stuff can just be intact on these nice and simple and easy and standalone, then, if you really want to use the jenkins stash plug-in or the unique report plug-in or whatever then ship stuff to the jiggy's controller and then do stuff in your drink, it's fine, but it is one of those things.

E

Sometimes, if you know jenkins, really well, you're, probably going to do things more in jenkins than and this one is finding the right balance and find the right tool for the job.

A

Yeah, at least it provides some freedom and if you ask jenkins experts heavy users, actually the recommendation is to do less in jenkins, literally, the rest presentation named like that by jc glick at one of jenkins walls 2017. I believe, because the recommendation, if you use common, build tools like maven gradle or whatever delegate more to these tools um and use drinks, basically as a proper as a specific integration.

E

I mean this is still very early in the detection. Plugins life I mean still suddenly just gone one zero, so we we still, I think, need to build up this kind of best practice kind of recommendations of how we think you should be using this, but I think I think you're right, like I think, the more we can push uh delegate operations to the underlying tools like gradle whatever, and that stuff tends itself more easily to techtone and let jenkins be the orchestra of those things.

E

So you know what do you need to do before techton and aftertechton?

E

One thing we could do with improving is um helping make it here to get state from the tectum pipeline to the jenkins controller, like in jenkins, we've often used stash and unstash, and those kind of things inside of a step because we've often had- and this is after I found this really confusing. When I first started on jenkins, you might have a pipeline and you think the pipeline's running on a node somewhere, but it's actually running on the controller and you're, never quite sure where to step.

E

Is it running on the agent or is it running in the controller where's the state and you've got to stash and stash, and it is a bit kind of confusing. It would be kind of nice to have very simple, canonical steps in tecton run that we can just say I finished my tekton pipeline now upload all of this state to the controller so that after we we go back to the jenkins vault we can, then you know use any of the standard junkies plugins, for I know jay unit reporting or something like that.

E

So it'd be nice to polish the experience a bit more so that after tecton has happened, you can use state from inside tecton um injectors and vice versa, but yeah, I think we're all gradually learning to improve.

A

In this space, anything back from users will be appreciated, so the plugin has been just released as 1.0. There are github issues. I believe everything will later show the links. So if you try it out, if you have any ideas, please use it as a channel for feedback, and it will much appreciate it.

A

So there is another question: is a build lock remained in jenkins, even if I delete a pipeline run resource from.

E

Kubernetes, yes, it's a quick answer. Yeah.

A

And that's a great answer. Thank you.

E

Although if you then delete the jenkins controller pod, the log is on a persistent volume, usually in in kubernetes.

E

So if the pod comes back and the same system volume is there, then you still keep your log, but it's worth saying that even if you're just using jenkins and so forget tech time for a second, even if you're just using jenkins, you should get configure your junkies to backup all of your logs and artifacts to long-term cloud storage like a bucket or something because you can use your jenkins controllers and if you're, not careful, you can use your persistent volumes.

E

I I've deleted clusters by accident quite a lot of times in my life um and often the positive volumes stick around, but they don't always so um yeah make sure you put everything in long-term uh storage. If you can.

F

James, do you want to talk briefly about the um the catalog stuff, the the jx catalog stuff? Yes,.

E

F

I don't have a demo for that.

E

That's quite okay: that's quite right, um yeah! So this it's a slightly long story. So I've been on a very long journey of the last kind of five or eight years of building and deploying pipelines across many repositories, and we started off in the early days with jenkins, shared libraries and then using ginger jet libraries, of course repositories and then gradually kept iterating and iterating. Then we got we started jenkins x and we started generating junkies files. Then we started to use text on.

E

Then we tried using tekton on the tool called kept with kpt from google, which is a way of sharing yamls across git repository, which is pretty cool, and we've been on this long journey of continually improving. How can we effectively share pipelines across repositories?

E

So let me step back for a second imagine: you're a company and you have 10 microservices and those 10 microservices all need pipelines for dealing with ci and pull requests and dealing with releases. If those 10 microservices are all saying, node or java, the pipelines are probably exactly the same. So you want to share those pipelines. You don't want to keep maintaining 10 separate, huge chunks of yammer, but sometimes one of those microservices wants to do one thing different.

E

You want to do a step before you release or after you release, so you want to generate some extra dock, so you want to do something different, um so you've got this classic problem. How do we customize a pipeline for one repository but maximize reuse across all of the repositories and not have a maintenance nightmare?

E

So we've been trying to kind of work on this kind of tricky problem for a number of years with various efforts in the changing sex community, the one we've come up with now is we borrowed a trick from um there's a tool called co ko in the go community for building container images and there's a tool called mink from the k native community and that's for again building container images and they use a little trick where they use the magic image tag.

E

They use a magic image tag in their case they do docker builds. So we decided why don't we use a magic image tag, so we can write canonical text on pipelines. It's 100 tecton, there's no dsl, there's no wrapping, there's no cogeneration, it's just vanilla tecton, but we can post process that tecton and expand magic tags. So we have a magic uses, tag on images which basically lets us reuse, a whole task from a catalog or just a specific step from a catalog.

E

Now this might sound a little bit weird and wonderful, but the basic idea is all of the pipelines. We use in jenkins x, they're all pipeliners code they're all stored in the git repository of the microservice, but we don't reference any specific image tags or commands in any of those steps. We just reference a version of a step in a library, and then this means we can then override any step in any repository to use a different version, a different image, a different command whatever.

E

So we can override any of the step method anyway, but at the same time we can share a definition across repositories. So it gives us this kind of dual um optimization of we can share. We can upgrade all the pipelines together without having to do pull requests on every single one of your microservices, but we've always got that get out of jail, free card that we can just change one step in any pipeline whenever we want and then once we're happy with it, we can then migrate it upwards to our shared library. So it's this.

E

It is a really thorny problem right in ci cd. How do you share pipelines everywhere, but use pipelines as code, but allow microservices the independents to do things differently and then over time align things, so you don't have a maintenance nightmare. It's a it's a super hard problem. Unless you've tried to solve this before you don't realize how really hard this is so to go long story short. We have this user syntax, which is just a magic string on your images.

E

We've added that support into the tecton client plugin for jenkins. So if your text techton yaml, has this uses syntax, um you can then reuse that pipeline, so any of the junkies x pipelines you can just use them inside jenkins via the detector plugin. The only one thing you need to do is in the jenkins configuration you need to enable the tekton catalog flag in the jenkins plugin, which basically then triggers this post-processing step that it basically looks at any of these users tags and then in lines.

E

Whatever that step really is so it's a form of reuse, you can think of it as kind of like jenkins-side libraries, but it's slightly different. It's based on referencing files and git, using a git tag, basically with a very simple override model that you can then override locally any of the values.

E

So it's it's totally different to ticket shared libraries, but it's vaguely conceptually like it is trying to solve a similar problem. The really the thing I really like now, you might find this usually stuff a little bit weird when you first look at it, but the thing I really really like about it is once you get your head around it, it's mechanical and simple: there's no programming, there's no weird functions to test. It's literally just sharing yaml and there's various tools, command line tools in the jigsaws community to visualize the effective pipeline.

E

That would be run if you use the pipeline, so you can kind of understand what's going to happen before it happens. Things like that. I hope that helps um if you raise issues on the jenkins tecton plugin, if you're totally interested in the user stuff or pop by the jenkins x slack channel, and we can just talk about it and I'll post that in the chat. um But it's worth remembering, none of this is necessary for the tecton plugin in jenkins.

E

You could just use 100, vanilla, jenkins, well, hudson, vanilla, jenkins and vanilla texture without the uses stuff, the usual stuff is completely optional, and it's up to you to use it. I posted on one of the chats earlier. By the way um I saw a blog recently about a company that looked at jenkins x and for whatever reason, decided didn't quite fit their use, so they end up building something very jenkins x-like.

E

They need hit this problem as well. How do we reuse pipelines across repositories allowing customization but maximizing sharing, and they went a slightly different route to the users approach? They went with customize, which is a kubernetes tool that lets you take some shared yaml and then customize it by overriding things, which is another way of doing the same thing. So if the user's thing doesn't kind of gel with how your brain works or your team customize is another option, I personally find customized quite complex.

E

I find it even harder to use than the easy stuff the easiest stuff once you get your head around. It is you're quite simple, whereas customize can be a bit complicated, but you know try both I mean use one or the other. It's worth saying as well. Sorry, I've been rambling about this for quite a bit tech time itself does have a reuse mechanism built in techton pipelines. Let you reuse a task, so you can reference a task in git, which is great the only downside with the task reuse in tecton.

E

Is you can't change it? You can basically just reuse a task and that's it and if you want to add a command line argument to the maven deploy step, you basically have to fork the whole task somewhere else and make a new version of the task. And then you have to reference a new version and then, before you know it everyone's got their own forks of everything. Everywhere you get in this kind of a big mess so that tecton doesn't yet have a step, reuse mechanism, which is what the user stuff does.

E

I hope it does at some point, there's a few pull requests and extension proposals in the tectum community to do that, and I really really hope texan gets it eventually. But right now uses is the is the best I've seen sorry for the long.

A

Answer for the details looks like we need another meet up about the jenkins antique ton, catalog configurations well good to be to consider for the next two months. Okay, uh so there are a few questions left. I suggest to firstly finish with the presentation another few slides left about the future of technology plugin, and then it will be natural to answer questions which we still have from the list and feel free to submit more questions.

A

While we are answering.

A

This part of the presentation.

A

C

Yeah, oh yeah, sorry I thought so um so, what's next for the tecton plant plugin itself, so right now uh we uh had an idea for the tectonic platform to have its own uh dsl for tecton, uh so that uh so that we can just use uh the pipeline in jenkins file and we started work on that and created uh the tecton create raw extension for that, and we, as you saw in the example that that we can use it right now, uh so the in future.

C

We are planning to extend this to or to do some other stuff and see what else we can come up with, and this was also an idea for uh g-shock and we are. We are still looking at how that's going um then. uh Could you talk a little more about avoiding taking up an executor, probably.

F

Yes, yeah sure, so um that's it's!

F

It's probably it's probably more of an optimization at the moment, but um when the pipeline, when we kick off one of these with uh jenkins, you kind of need to run it through the pipeline order on an executor um because it clones the repo and it has access to the workspace, um we're we're investigating ways of trying to not do that. So we can minimize resources as much as possible on the jenkins side um because it executes another.

F

You know jvm somewhere, that is running and we want to keep that, keep that as lightweight as possible, so that um we get better build densities in our clusters. That's that's the general idea.

C

Then uh the next one is support files with multiple types. So, right now we support yaml. So in the future we hope to support json properly and also are we thinking of supporting xml.

F

I think I think it's also having yammels with multiple documents in the same file. um We currently, we currently sort of say don't do that, um because it makes it difficult to understand what if there are multiple pipeline runs or a pipeline run and a task run in the same file. um What logs should we be streaming to the console it sort of makes it a bit awkward?

F

um So we we, we say, don't do that at the moment, but it'd be nice to be able to handle some level of that, um and that probably leads on to the next point, which is like the naming conflicts between the two. So the demo I gave uses the generate name feature in kubernetes to always get a nice unique name, um but there are times when you probably possibly don't want to do that.

F

You may want to name your pipeline, something a bit more predictable um and similar to sort of the the the jx um way where we actually name it based on the branch or the pull request number and the build and build id as well. So that might be a something that we would look at.

E

If, if you enable the user's syntax or the protection catalog, the name is based on the repository and the branch and appear and everything but yeah. We could maybe do that natively for all texan pipelines. Maybe.

C

We could probably also uh use like jenkins x in the back to kind of uh like what you're doing right now, the effective pipelines- that's uh like that's a really good method to get context of the same context, all over the pipelines yeah. So currently we do our tecton catalog integration that way, and we also have to figure out how to how we can make that better.

C

um Then, after after that, uh there is a gsoc project idea on cloud events plug-in right now and in the future. uh Once that rules out properly, uh we will think of how we can get that integrated with the tecton plug-in and maybe uh you know, be able to trigger tecton or tasks like through jenkins through cloud events. So that's also something I think about or if it's, if it means that we directly trigger protect on resources through problems and not through jenkins itself. So this is also something uh something we are working on.

C

So uh this is what's next and we uh we we discuss all these things in our cloud native sig meeting, which happens every friday. um The meeting is up on the jenkins event, so calendar and you can check it there and thank you for joining.

A

Thanks to all the speakers for the presentation, thanks, girls and sweetheart thanks james, it was a really interesting discussion and yeah a lot of context. So again we are looking for feedback from all participants. We encourage everyone to try out this plugin and some feedback through github issues, and there are more demos and relief presentations coming soon, hopefully so stay tuned.

A

um There was one question regarding the roadmap, uh so if we could return back to the previous slide, so the question is about: is there a plan to add the github's patent support for the text on plugin.

E

That's a good question: I'm not sure the answer, though well one way of looking at it is the tecton plugin can use pipelines in gates and trigger detect, so you could store detection pipelines in your git repository and then use git ops to manage the pipeline.

E

That's a slight different thing to when people talk about githubs and kubernetes, where they're using a git repository to do the cube ctl apply effectively to deploy applications, um I'm not sure if the tecton plug-in is the ideal thing. For that I mean you can obviously use that um you could make a diy github solution with the title plugin, but people normally think of tools like jenky, texas or flux or argo for for the last mile.

E

Google config sync is another tool that suppose fleet is another one. There's a there's a bunch of tools like that. I think more. The text on plugin feels more about doing ci and releases rather than managing prediction clusters.

E

You probably don't want to jenkins serve in your production cluster, just to be able to deploy your applications.

F

Although you would be able to use the the published terraform images, which is something that we can't do at the moment, because they don't have a shell, um so you would, you would be able to create a pipeline to do a terraform apply, for instance, um onto an environment.

E

Yeah there is the what makes production you need something like a cia environment or something to run your terraform to make your cloud infrastructure, so the jenkins plug-in would be ideal for that, um for you know, defying your terraform and stuff it'll be ideal. Yes, absolutely right!.

E

It's a slight tangent, but often the githubs approach that, like we've works, have been popularizing with things like flex and again. Why not?

E

It's a slightly different use case to terraform like terraform, you tend to run it once to set up your cluster and then you might run it occasionally if your infrastructure changes, but it doesn't have to change that much but to deploy new versions of microservices, which happens very frequently and very rapidly.

E

It's fairly common to run a deployment agent in your cluster to avoid having to publish through access production, cost admin tokens around the internet or whatever it is. So um I'm not sure if people would want to run jenkins inside production clusters.

E

um However, using jenkins to run your terraform is a perfect example with the titan plugin, I think that'll be awesome.

A

So thanks for the answer, the next question, there are many stages in pipeline orion, like docker, built et cetera. Then your way to showing the stages in jenkins ui. As I understand, we only see build stage in jenkins ui, which uh you might implement in jenkins file. So what about the tipton stages, steps, etc?.

E

We could totally do that. We don't right now, um but we could talk.

E

A

Isn't it on the roadmap for the plugin.

C

For like the users to see, maybe if we can raise an issue about this and then work on it later on, it will be interesting to see the tecton steps and everything like in the jenkins uh ui. It will uh like so that the user doesn't have to go back to the ammo and then check this the stuff over there. They can see everything in a nice graphic.

E

Yeah it'd be nice to basically make make it look like you know, blue ocean, like a full pipeline, with steps to where we're mostly just showing the logs right now it'd be nice to publish the pipeline graph.

A

Jenkins has api for that um and it would build a good uh story for pluggable uh log storage, so you might have seen that we already have implementation. For example, you can store pipeline locks in aws cloud watch. You can store them in other services and there is api under the hood, which allows to implement other connectors and potentially we could stream pipeline execution data from other services to jenkins and display them as natural pipeline graph steps.

A

So architecturally wise. We have it. It's definitely something interesting for ticket on and maybe for many other stories like, let's say, jinx file runner we were discussing recently but yeah.

A

As we have said. Please send me the ticket if it doesn't exist already, because it's definitely something to discuss for this particular plugin. Something.

F

Else that I actually failed to demo was um we support the checks api as well. So when the pipeline runs, you get events going back to, for instance, github, um if you've got that, if that's how your your repository is configured and that works really nicely so we could add in, um we could add in support for more information going back onto that, so um put informations about the the tasks and the task runs um and potentially log um back into that api. That's also an option.

A

Just to clarify that is engine use, uh checks, api plug-in jenkins, or does it use tickton implementation? It uses the checks api plugin in jenkins, okay, so basically you can stream uh uh these events to any receiver, not just github actions, but, for example, uh there is a prototype for slack integration, etc. So any consumer of these check events can be used.

F

Yeah at the moment, at the moment, we're only really dealing with like pipeline is running it's completed or it's failed, but we could certainly add in more areas around that. That's great.

A

Thanks for the qualification.

A

um Yeah there is another question: uh pipeline resource was striked out in one of the slides. Is it removed from tikton.

F

So my understanding is, it's still kind of there, but it was left in an alpha state. It wasn't promoted to beta with the rest of the tecton resources, so um I know it's like it. I think they're thinking of it more as an implementation detail, rather than something that you actively go and deal with.

C

It could come in handy where, in the situation where uh you're using you want to use tecton as well with jenkins, uh there are some things jenkins could do, uh which tecton can't and you have a probably have a lot of stuff set up.

C

But you would like to start using tektron, uh so you can use the plugin to kind of like, instead of switching between them uh continuously, you can just use the plugin to kind of manage everything from jenkins itself, which would be which would be nice so, uh but it depends on, like use case use case. So if you keep switching back to techron, because you have a lot of stuff over there and not much stuff on jenkins, so it makes sense to just use techno.

C

But uh if you only use a little bit of tecton with everything else in jenkins, so it makes sense to use the plugin at that point. So you don't have to keep switching between the electron dashboard and the jenkins dashboard.

A

Thank you. So the last question we currently have in the list pipeline resource was right. We already answered that so there was another question but, uh like james answered it asynchronously. So uh thanks a lot to everyone. Again, we encourage everyone uh to send feedback to try out this plugin. uh We will be also sending a follow-up feedback form after the meetup previous links to recording slides, and we will ask a few questions.

A

So we, if you have a few minutes, we will appreciate if you fill in this feedback form um and yeah, and any additional information will be appreciated.

A

Like we said uh in uh the discussion, there is cloud native physique meeting uh happening in just two hours. So if you want to join and to have more discussion, you can join this meeting and they will be rather technical dive. So any hardcore questions are welcome there and you have two hours to try out the plugin. If you want and then come and ask any questions, uh okay and yeah right now, we will start.

A

We will stop the recording, but you're welcome to stay online and again we will have something like 50 minutes, for you know on formal informal discussion, any topics about tiktone jenkins integrations or basically, whatever other topics you would like to bring up in jenkins, you're, welcome to stay and discuss these topics thanks. Thanks again to everyone and looking forward to meeting you on may 27th, so we will have an online meetup about uh jenkins kubernetes operator, which is another ongoing project which provides support for managing jenkins using a classical literature solution and yeah join us.

A

So thanks all any closing notes from.

A

Others: okay, all right him, andre james garrett,.

A

If no thank you and yep stay online, thank you.