Continuous Delivery Foundation cdCon 2020, 20 Oct 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Conditional Constructs to the Rescue - Priti Desai, IBM & Jerop Kipruto, Google

Description

Conditional Constructs to the Rescue - Priti Desai, IBM & Jerop Kipruto, Google

What do you do when your CI/CD use case is a bit more complicated than a simple build, test, deploy? What about when you want to build from a GitHub repo but only for a specific branch, deploy only if an image exists in the registry, or cleanup even when things fail?

Being able to express your CI/CD pipelines as a directed acyclic graph (DAG) is powerful, but when you want to add conditions into your pipeline or take action even when things fail, it gets more complicated. In Tekton, execution order has historically been controlled by expressing dependencies between tasks in a pipeline, or by explicitly specifying the order.

In this session, we’ll take a look at how to express conditional behavior across several major CI/CD tools, and take a deep dive into how to use conditional constructs with Tekton:

Efficiently guard execution of a task with When Expressions
Reimagine the DAG with Finally

A

Welcome everyone uh we are going to present on conditional constructs to the rescue. uh My name is preeti desai and I work for ibm.

B

And my name is rob kipruto and I work for google thanks gerald.

A

uh So we are going to talk about techton and conditional constructs on uh in tecton. uh So, let's see what is ticked on uh techton is. If you google, for tickton, it shows you as a tooling company, but wait. Are we touring company? um We are not. Actually we are so we are a tooling company, but we build uh tools. We build framework uh to facilitate uh common, to facilitate cicd use cases or ci cd pipelines. So basically it's a software tooling. um So, let's, let's see what is the tecton pipeline?

A

uh So techton pipeline is basically a collection of tasks and uh techton pipeline. uh We represent any cicd use case. We represent using tecton pipeline now think beyond a sequence of tasks. uh Tecton pipeline is it's a one of the core building blocks of tecton and uh it's being a collection of tasks and each task actually executes a pod on a kubernetes cluster and in programming paradigm. A task is a kind of a library which can be shared across multiple pipelines.

A

Task is this collection of steps. Step represents container images. Basically, it can have multiple steps and can run multiple container images, uh running specific, tooling, um on set of inputs and basically produces outputs.

A

So uh this this looks good, uh basically pipeline being a graph uh here. Scheduling is a key now because it's not a list or a sequence of tasks. um So even all the tasks runs fine runs successful. Basically, everything gets executed.

A

But what, if something fails a pipeline needs to decide what needs to be done next, so to kind of help. With this scheduling um we have implemented designed and implemented. Few conditional constructs and those conditional constructs helped uh solve common cicd use cases. For example, uh you know control or guard a particular task, uh depending on what the prior processing looks like uh whether something uh says that okay go ahead and execute or not so basically, uh one of the con. The conditional constructs helped us uh design the these kind of uh use cases.

A

The other uh common use case is, we want to design a generic pipeline and utilize in many different environments or regions or cloud providers, and one more key use case is uh execute something before exiting the pipeline, uh and so if there is a single failure, but we would like to execute something even beyond that failure.

A

So here we have a conditional constructs to our rescue. uh Now next jerich will take you to the next.

B

Slide you pretty for this next slide and content. We are going to explore exactly how conditional constructs have come to our rescue by looking at how we build the tecton pipeline and then, secondly, how conditional constructs have come to our rescue and particularly looking at when expressions in tecton and finally tasks in tecton.

B

Thirdly, we look at a demo um to demonstrate some of the use cases that we'll talk about and, lastly, we'll look at what's next um in tecton pipeline conditional constructs so for standard thcd use case, it usually involves building an image then running end-to-end tests and finally deploying that image in tecton.

B

We have a task to clone a repository into a workspace, and the workspace is a shared persistent volume among shared across all the tasks.

B

Secondly, we have a task to build an image based on the workspace and publish it through an image registry. Thirdly, we have a task to run end-to-end tests using that image and, lastly, a task that um deploys that image to dev or staging environment depending on use case. Additionally, you could have a task to deploy that image, a particular region, depending on the use case when we have this pipeline, then in tecton we have a pipeline, runs that insane this pipeline and report success of failure based on the execution of each task.

B

So if all the tasks are successful, then the pipeline run reports success and if any of the tasks fails, then we will. um The python run exits on the first task failure.

B

So what you provided in tecton is one expression and particularly, would look at how pitch uses parameters and, in this use case, to build or enable us to create a generic pipeline um to be used for multiple regions.

B

So in the previous example, what we had is that it was just deploying to one region, but in this case we could be in a situation. We want to deploy two different couple of regions, so one expression allows you to specify an input operator and values to be able to do this and have a generic pipeline, um and when we can do this for, uh like region parameter, probably we could also pass in cluster parameter or even cloud provider parameter.

B

So for demonstration purposes, though we will look at uh particularly the region parameter. This, uh I would have to say, was possible without an expression previously. Well, we used to use condition crd, but when expressions bring simplicity and efficiency without wasting the resources used to spin up codes for evaluating checks.

B

So if we look at this example- and we are want to deploy to particular region, then the flow of execution would be as follows, and you can see that one of the deployed has escaped while the other one is executed.

B

uh So what? If we're looking um at a situation where the deciding factor is not a parameterized string and the some kind of automation to compute a value that needs to be used in when expressions?

B

So what happens in this case is that you can use task results and have them substituted in one expressions and used to evaluate whether to execute or skip a task.

B

In this example, czech repo produces results, exists and region which is then used to guide, build image tasks and they deploy tasks that we have at the end. So in the case where the file exists- and we want to deploy to us, this will be the flow of automation.

B

As you can see here, then everything runs to green, and it's only the floor is that is kit, um but when we're in a situation where the file is missing and um therefore exist is set to no uh you'd find that build image is kept and everything afterwards is kept as well.

B

Finally, we're going to look into how finally tasks are used to even provide more conditional construction capabilities and we'll have pretty talk about that.

A

All right, thanks cherub, so, uh let's um so so far, we saw how a task can be guarded, basically guard a task in a pipeline and also design generic pipeline for different environments using one expressions.

A

Now, let's revisit the graph we saw earlier in the presentation uh now here it the scheduling here. Basically, everything is green. Everything is successful, it looks great, but um what if something fails? Basically rest of the tasks are skipped. They are not run, they're not executed, but there are a lot of cicd use cases in which we would like to have um something basically something running even beyond a single failure.

A

So uh we actually uh so, and one of the common use case is basically the cleaning up, irrespective of success or failure. We would like to clean up our resources, so here uh we to kind of uh help us implement this kind of uh functionality.

A

We implemented something called finally tasks. So now we have two sections: we have dhg tasks and we have finally tasks. The entire graph runs. uh If something fails, what happens is basically the? Finally tasks are executed, the dag task, everything runs. Everything is successful, but we still run any anything that is specified under the finalists section.

A

So basically it guarantees that anything we have specified under. Finally, section is getting executed, so here, for example, um everything green finally is executing clean. It clean up, cleaning up is done, but which is not the case right. Hardly it's the case in real world. So here it shows that you know something failed now and integration test failed. So we skipped the deployment, but we still run cleanup now here the key is the uh exit status. What happens is one of the dag tasks failed. uh The pipeline run exits with failure.

A

Let's say if everything was successful, uh dhd task runs all the dag tasks run fine successfully, but one of the final tasks failed uh pipeline run would still exit with failure, because it's a one of the tasks in the entire pipeline, even though it's a different section, but it's still part of the same pipeline so finally does influence the exit status.

A

uh One of the other main use case or key use case is sending notification are kind of so here the idea is. We would like to kind of get some information based on whatever the dhe tasks.

A

Are they execute and then they produce some results. We would like to communicate that result to finally task so this feature was uh introduced to help kind of uh let's say a projected approach. Project is created and that project name was basically needs to be communicated to the finale task. So here it is this project here. Is the project now go ahead and clean that up? Our notification here is my pr number and basically send notification to this pr, author and here's the number so uh so.

A

Basically, communication is the key and that's why the results or any processing or any output from the dag is communicated to. Finally, so uh let's look at demo all these use cases we have kind of. uh We have put together a demo to show you the deployment and also uh basically how it looks on dashboard and also uh tecton cli.

A

So I'm going to play this here. uh This is the github repo that I have all the pipelines, all the simple sample pipelines uh stored, and here we are running deployment, so our first use case is being deployed. Let's look at uh on the dashboard what's happening so here we are cloning, the repo the cloning is done, and this is the repo that was cloned.

A

Now we are building an image which has basically a sample hello, world kind of application in golang.

A

um Now, once this is done, yes go ahead and apply it because build was successful, so deploy okay, we are deploying this image and then we create a service on this port, uh expose this port so that we can access it now, basically go and basically access that application and see if it's.

A

Running okay, yay! So now we have that application running and you can notice the time uh local time here and yes, hello, siri. Oh, let's look at the other next use case. uh Deploy in a specific region. Deployment is happening. We have a pipeline designed for keeping three different regions in the pipeline, but we would only like to deploy in one region clone it from the github url build an image.

A

Now deploy but deploy only in one region, and here is the noteboard.

A

So now let's go so it says the tasks were skipped because, of course, the regions were, the tasks were not deployed. uh So here the tecton cli. It's basically, let's describe that pipeline run and says we would like to uh deploy in east and then, let's see, we should have east and central as skipped.

A

A

Yes, so our task for deploying in western central west and that application in the east region is accessible on this port.

A

Okay, so now, let's move on to the next uh use case here we are kind of guarding uh tusk. Basically, we do not want to build if something uh fishy. We see something fishy, for example, or something some uh so here we are cloning and we are running our check.

A

Yes, clone from that repo and run our check. We actually found a bogus file. So if we find that kind of file, then yes, we basically exit out. We do not want to deploy, build or deploy. So now, let's check. um Yes, it was finished and what happened so we described that pipeline and we see that only two, basically the clone and the check is successful.

A

Now we would like to see whether.

A

The rest of the pipeline tasks were skipped.

A

So build and deploy up image. Please script. Okay! So now next use case uh something fails in dhg tasks. So here we are deploying this particular pipeline.

A

And look at on the dashboard we're cloning.

A

Yes, now building an app image once this is built, we run end to end tests.

A

So here we have a kind of fake end-to-end test, resulting in failure, and so now yes, the building was successful, run okay, so we exited with one. So it basically resulted in failure. So deployment must be skipped but clean up. We want to clean up our uh workspace, the persistent volume which was shared so now clean that up and then it basically says that the pipeline resulted in failure.

A

So here yes, that pipeline failed.

A

And we described that pipeline.

A

Yep, so here you see the um reason why it is failed and was: were there any skip tasks? There must be one.

A

Okay, so deployment, so basically, this kind of shows the overall uh use cases and how, when expressions and finally are being useful um here so.

A

B

Thank you pretty so, with what we've seen right now, uh we've looked at when expressions independently and finally tasks independently, and we've also seen that. uh Finally, tasks are executed, regardless of success or failure in the pipeline.

B

What is coming up next is that we're looking into having one expressions in finally tasks to allow for conditional execution of finally tasks. An example. Use case is what we have here, for example, if when uh end-to-end tests or integration tests fail, you may want to create a github issue, and in this case, then we don't want to uh create the issue when it's successful, so we can have one expression guiding finally tasks, which is a feature that is coming up soon.

B

Additionally, something else that we're looking at is finally task time out. We want to have a separate timeout to ensure that finally, tasks actually execute because currently, which is it, uses the same timeout as the ddg tasks and looking at a situation where, for example, the integration tests run for too long and the pipeline times out, then the final tasks would be executed.

B

But when we have a separate timeout for uh defining tasks alone, then this will ensure that they actually execute, even when uh the other parts time out with what we've looked at, uh we can then use one expressions and final tasks to create powerful packing graphs that can solve many complex clcd use cases to try this out.

B

You can check out the one expressions documentation in tecton pipelines, repo, as well as the finite task documentation and we've also provided these pipelines with finally tasks and when expressions on a repo that you can access and also play around yourself. Thank you so much thank.

A

You hello, everyone. Thank you for joining. Thank you for joining the presentation.

A

Hello, hey jrf,.

B

um Yeah and we just also added uh more references um to the linked uh slides in the event itself, beyond what we had in the recording.

A

Yeah yeah and the slides are available on the uh schedule. Basically, so you can access it from there um and there is uh so. I would like to thank bits of info. uh He has posted his github repo. uh He has a very uh in-depth knowledge of the uh you know the whole cicd state manager with techton. So I would like you to kind of look at his repo and there was also a question in the chat about the converter.

A

uh The question: let me read it: how can we convert a freestyle jenkins job to a pipeline job? So the answer is there is no direct converter. You basically have to understand the design, principles and kind of uh based on what job you're trying to convert apply it uh to tecton like.

B

Yeah and in the repo as well, we have like end-to-end examples that can demonstrate how these features can be used right.

A

And any more questions, anyone, uh yes, it's not intimidated by techno. Basically, yes, you can do a lot of and it's still evolving. There are a lot of a lot of great chairs already in line um and it's actively being developed and it's a great community, if you guys are interested. uh I've worked on open source from basically since past five years. I started working on technon last year and the community is amazing.

A

It's very warm welcoming anything you do is appreciated, so feel free to kind of, and the jared posted a link to the slides. uh The references section has details on basically how to get started with tecton and the repo and community repo.

B

Yeah and finally towards pt said that and the community is very welcoming and you can get involved, there's a link that I just also added. There talk me into repo with more details on how you can participate in helping us shape. Tecton.

A

You therapy you're in new york. Yes,.

B

A

B

A

California, we're on two different sides.

B

Yeah, um I'm wondering people in the chats. Where are you coming from? Have you used conditional constructs before which tools.

A

B

For your cicd systems.

A

Yeah feel free to post it on chat. If you guys have used what kind of ci cd system you guys are using and what kind of was your experience happy, not happy yeah. I wish there was something like in a hand raised thing.

A

Where you can just raise your hand, okay, so bits of info is saying, used to do some jenkins and now tucked on all the way, great yay.

A

Okay, sergey is saying we use spinnaker, but also using techdown, to extend some custom workflows. That's great.

A

Yeah, it's a fairly new project and a lot of uh a lot of uh move happening and it's great.

B

Yeah, maybe christy, I don't know, maybe you can share more about like your, how long you've been with protector and how what you've done with the project. So far since you've been there longer.

A

Yeah, it's been an amazing journey with tekton. I started last year october, it's been a year now and I'm one of the owners in tecton pipeline, um and there has been so many uh great features that I could basically be part of myself and started with talking to christie. She was awesome and just getting into community uh started attending working groups and basically, as I I have, uh I used to work on serverless.

A

So that's where my experience coming from using tecton and we were missing couple of features and that's how I started getting involved in tecton and kind of took it off. How about you gerald you're.

B

Not I'm a new one, I'm a much newer contributor. As you know, I just joined um much this year, but like we don't bodies, I've officially started contributing in nfl, but as pretty sad, it's a very new project. There's a lot of things to be done and the community is very welcoming and supportive. I've been able to um work on big features, even as a new contributor in the project like when expressions in itself, so um yeah I'm really excited about the direction that protection is going.

B

We have a question from paul about how we are integrating the two um so currently, when expressions are not supported in final tasks, but um as you, as you saw, there's something that is currently in the works is uh supporting results in one expressions and no supporting results in finally tasks um like results from uh from tasks that were in the ddg and once that is supported, then we can also um introduce one expressions in final tasks so that they can execute based on those results.

B

So this is all actively ongoing work, uh particularly supporting results, infinite tasks and then the next step will be having one expressions there to guard whether final tasks you'll execute or not. I don't know if pretty has anything to add.

A

Oh, no, we are out of time actually so yeah yeah, okay, so yeah. Thank you, everyone for joining and have a great rest of the session. Okay. Thank you.