Continuous Delivery Foundation cdCon 2021, 6 Jul 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Pipeline Templating: Enabling DevSecOps Adoption Across the Enterprise - Steven Terrana

Description

Pipeline Templating: Enabling DevSecOps Adoption Across the Enterprise - Steven Terrana, Booz Allen Hamilton

Within large organizations there exists a tension between the business's need to achieve economies of scale and the local development team's desire for the autonomy which enables innovation. Many organizations struggle at finding the balance between adopting standardized approaches to continuous delivery and expecting each team to be responsible for creating their own bespoke DevSecOps pipelines. In this talk, Steven will demonstrate how the Jenkins Templating Engine (JTE) turns this classic "either or" problem into a "both and". JTE empowers organizations to create truly tool-agnostic pipeline templates that can be shared across teams enabling effortless scale without sacrificing team's ability to innovate.

For more Continuous Delivery Foundation content, check out our blog: https://cd.foundation/blog/

A

Hello: everyone thank you for joining pipeline templating, enabling devsecops adoption across the enterprise. My name is stephen tarana, I'm a chief engineer at booz, allen, hamilton and today we're going to talk about a approach for implementing a large-scale devsecops pipelines.

A

If you're like me and you've worked on implementing pipelines from multiple teams at the same time, you've found that it's really challenging and and the reason it's challenging is because different teams use different tools and all the ci tools in the industry are really focused on how to implement ci cd pipelines for a single application.

A

So you'll have a jenkins file in every source code repository or a git lab file, and each repository- and you end up copying and pasting that for different teams, there's some work. You can do around shared libraries, so try to consolidate code. At the end of the day, you have multiple source code repositories.

A

Today we're going to talk about how the jenkins templeting engine allows you to implement a single tool. Agnostic pipeline template that you can share across teams and we're going to do that purely through a live demonstration. So, there's no slides that accompany this. We've got vs code and a browser and we're gonna dive right in so first things. First, we need to get our local development environment set up, so I've got a just file. This is just file. Has a couple helper scripts: it's going to build a docker or a container image.

A

That's using jenkins lts! It uses jenkins configuration as code to configure the jenkins instance, uh and we've got some other helpful automation in here. uh So I want to get right to the demo, but if you're interested all of this is available online uh in a github repository that I will, I will share the link to so. Let's kick things off. I can run just launch jenkins.

A

That's going to pull the latest lts image and build that custom container image for us. That's going to make jenkins available on localhost 8080. So if we switch over to a browser here and go to 8080, we'll see that jenkins is about to start.

A

A

There we go so while jenkins comes up, let's go uh back to a browser, so the way, the way this local demo works is we're going to have what are called libraries. These libraries live locally they're mounted to the container image so that, as we make changes to this local directory jenkins is automatically able to pick up those changes within the just file. We have a just watch recipe that uses watch execs so that every time we make a change to a groovy file, it's automatically going to commit that to this local git repository.

A

So if I run just watch we're now watching and changes are automatically gonna get picked up. Going back to our browser here see that jenkins is now up and running. uh First things. First, is we need to create a pipeline job, so we're gonna? Stick with uh you know a pretty basic demonstration just to get the concepts of jte across the simplest way to make a jte pipeline is to create a regular old pipeline.

A

Job jte does support consolidated pipeline templates for github organization, folders or multi-branch projects, uh but using the ad hoc pipeline job is probably the the fastest way to get up and running for a demo. Let's make this a little bit bigger.

A

So if you, if you, if you have used jenkins before you'll you'll, be used to seeing the the pipeline script and the pipeline script from scm, when you install jte, it adds a new option uh called jenkins sampling engine. We can define two artifacts right from the console. One is called a pipeline template and one is called a pipeline configuration so the whole theory behind the jenkins sampling engine. So it allows you to decouple the business process of your pipeline.

A

What should happen and when and what are the steps to the different tools that might be used across an organization to implement those steps right so before we dive into how jt does templating? The first thing to understand is that a pipeline template in jte is just a pipeline, just a regular old jenkins file. So there's nothing stopping you from from writing some pipeline code that you may be used to.

A

So we can just say echo hi world and we don't need a pipeline configuration because we haven't, you know loaded anything into this template, so we can click save and just to prove that this is exactly like any other jenkins file. We'll run this build and we'll see that in fact does pass so jte does support declarative pipeline syntax. For today we're going to stick to descriptive, pipeline syntax and I'll. Show you why um so, let's say we wanted to create a pipeline template that does a build.

A

You know this might be a build from maven. This might be built from npm gradle docker.

A

There's all these different build tools for different types of applications, and traditionally that would mean writing a new pipeline right for each of those applications.

A

So you might do something clever, like have a global library that has different implementations, and you know you have to import that library and then call a common entry point, um but you still have to duplicate your pipeline definition and make some tweaks across every repository to make it work. So let's show how our very first pipeline step. We can create a build.

A

So, alongside that pipeline template comes the pipeline configuration file uh in you know the jte 201 video. We can talk about how these pipeline configurations are actually hierarchical. You can create a jenkins wide pipeline configuration for jte and then add configurations onto specific folders, all the way down to individual application source code repositories.

A

So you can create governance hierarchies in jte that are as complex as your organizational hierarchies and there's some rules around how those pipeline configurations are aggregated for the sake of today's demonstration, we're going to stick with a common or a single pipeline configuration.

A

So we wrote a template here that says build so now. We need to actually make that mean something so in our pipeline configuration we can create a libraries block. Just to note this. This pipeline configuration syntax is just a custom, groovy dsl. That looks a lot like you know, a blocked, nested structure here, so we can create a libraries block and let's say that our team is using maven.

A

So we can save that and we can run it and right now, that's going to fail and it's going to fail because we haven't actually gone and created a maven library, so just to just to prove that we get some errors. The maven library hasn't been found. So let's create it. If we head over to vs code, there's a nice helper function here in our just file, or we can say just create maven and that's going to create a new directory for us within the libraries directory called maven.

A

Every library and jte has three main steps to it. You've got a resources directory a source directory and a steps directory, so the resources directory is where you can stash files that you want your steps to be able to consume things like shell, scripts or python scripts or json files. Or what have you the source directory is where you would define classes.

A

There are some use cases where being able to have classes is, is very useful for today we're going to live in this steps directory. So back in our template, we said that we needed a build step and we were going to use the maven library to do that. So I can create a new file and call it build.groovy.

A

The base name of the file is going to be the name of the step that gets loaded within this build doc ru file. We can implement a call method and we can say println, building from maven right and we saved that the file watch that's running in the background committed it to the local repository and now it's already accessible to jenkins.

A

So if we rebuild this jenkins job, we'll see that jte tells you everything that it does. So you can see all the pipeline configurations that are added we'll see that we loaded a maven library.

A

uh There's a warning here, because the the method build is actually a jenkins step, so jte lets. You know that you've created a library step, that's going to override a default. You know regular old pipeline step and then at the bottom we'll see the jte puts out a log message that says we're about to implement the build step from the maven library, and then we have the the functionality that we expected to see so quick note on. Why is it a call method?

A

So that's just groovy syntactic sugar, um so in groovy, when I say build that's equivalent to saying build.call, so I like to take away as much of the magic as possible. There's nothing special about a call method. Besides the fact that it's some syntactic sugar in groovy that allows us to use the the shorthand notation for invoking the step.

A

So let's say we had another team and that team was using gradle, for example. So let's save these file contents and let's, uh let's create a new gradle library, so we can say just create gradle. It's going to spit out a new directory inside the steps directory.

A

We also want to implement a build step, so we can paste the same code in here and say that we're building from gradle. So if we go on back to jenkins, if we wanted to use the exact same pipeline template which traditionally would be stored in a source, a central source code repository and inherited by multiple teams, you would not have to touch this pipeline template to apply it to different teams using different text decks. This pipeline template that you create essentially defines an api.

A

You know, interface between the template and the libraries, so one team could have a pipeline configuration that uses maven and the maven build step is going to be used in this template, but another team could come in here and all they would have to specify. Is that they're using gradle? So without changing this template at all, which is common across multiple teams and just specifying a different library being loaded?

A

We can now rebuild this pipeline and, as you might expect, it loads the build step from gradle and we get different functionality, so that might seem pretty pretty trivial right, because we we've got four lines of code total here and we created two pretty small files, but the idea is is pretty huge right.

A

I've been on teams before jte existed, where there were something like 50 or 60 different microservices that each had their own source code repository and making a change to the pipeline would mean opening pull requests to every single uh source code repository trying to orchestrate those changes with the development teams uh sort of tweaking those pipelines for the different tech stacks and what jte allows you to do and what we just demonstrated is taking that jenkins file from every source code repository, making it tool agnostic and defining it in a central location.

A

So all that needs to be defined in each source code repository are the aspects of the pipeline configuration that are unique to that team because of the the hierarchical nature of the pipeline configurations, common uh common controls, like which libraries to use, if, if they don't change between teams, things like you, know, container image scanning that have to be consistent across everyone.

A

You really have the flexibility to control what pipeline configurations are governed and what pipeline configurations can individual teams update? So, let's, let's dive into a couple more features that jte has to offer.

A

So these libraries are only, as you know, valuable as they are reusable, so different teams are going to have different versions of gradle they're going to have different code quality thresholds. Maybe they don't want to enforce unit tests.

A

There's always you know nuance to how you would want a pipeline to work for a given team, so the way jte allows you to handle that is to pass in uh live library configurations.

A

So in this example, we can assume that the gradle library, for some reason, takes in a message field, we're going to say a from gradle.

A

So if I'm a library developer- and I want to consume that library parameter- that's been passed in, we can say the message is, and now we can access this config variable, and this config variable is auto wired to the steps at execution and it's they have access to the library configurations.

A

So I can say, config.message and save this head back on over to jenkins, rebuild it and we'll see that when the pipeline runs it's able to consume those those library parameters, so those can range from anything from what version of gradle am I using to you know what are the the targets that we want to execute? Is it gradle clean package greater clean, verify test?

A

What have you so an important thing to understand is that libraries and steps do not define method parameters right and the reason that they don't define method parameters is because going back to our template for a second.

A

If we had passed to the build step hi from gradle now, every single step, implementation of build across all the libraries would be expected to consume a string parameter right. So the reason these library parameters are so helpful is because it maintains the interoperability of the templates. We can just call a build.

A

These builds don't take parameters across all the different implementations, whether it's npm maven gradle. What have you and we're able to pass that information in the pipeline configuration file instead?

A

So another neat thing that you can do in jte is define application environments, so the one exception to the rule for not passing in parameters is when you want to create a deployment step. So, let's create, uh let's have our template, say, deploy to dev and deploy the test, and let's assume that we're going to create an ansible library, that's going to handle that deployment.

A

So this dev variable and this test variable are going to come from the application environments block. So I can define a a dev application environment and a test application environment inside the dev environment. I can define whatever environmental context. I need to to be able to do deployments to different environments.

A

So maybe I have a different kubernetes cluster for every single application environment. For our dummy example, let's assume that ansible expects there to be an ip addresses array where we can specify a couple. Ip addresses we'll do the same thing for the test.

A

Environment, all right and now, let's go create that that ansible library so we'll we'll save this configuration we'll head on over to vs code, we'll say just create ansible inside our template.

A

We had created a deploy two step, so we're gonna create a deploy to groovy file inside the steps directory, we're gonna, say, void, call and we're gonna take in an application environment. And now I can say.

A

Apm.Ipaddresses.Each and we can print out the the application environments from ln. You know deploying to.

A

Ip right and, let's also add a stage just to be able to visualize the the deployment that's happening, so I can say, deploy to apm.long name so application environments have a couple.

A

uh You know fields on them that let you differentiate, which application environment you're talking about so here we've got our steps. I can save this. We can head on over to jenkins and we can give this a run.

A

So here we can see that the gradle step happened. uh We've got a message from the jte framework that says that the deploy to step from the ansible library is about to be executed.

A

We have our deployed dev stage where we're using the ip addresses from that dev application environment and we have the test phase where we're doing a deployment to this test with these uh the test configurations.

A

So these dev and test variables- and these steps are examples of what jte calls pipeline primitives so pipeline primitives are just things that you can define from your pipeline configuration that automatically get created and injected into the pipeline runtime, and that's really how templating is enabled.

A

So we said at the beginning that these pipeline templates are just jenkins files and that's true, but what enables the templating is the fact that, prior to execution, jte parses your pipeline configurations, it loads the libraries and creates the steps, and it looks for any other pipeline primitives that have been created and it injects them into the runtime. So there's actually an object called build in an object called deploy to and dev and and that's how templating is enabled.

A

So there's a couple other primitives that you can create that we may not have time to get into today. So there's things like stages where you could group multiple steps together to keep your templates dry so under stages. I could say that there's a ci phase that does unit test, it does static code analysis and it does a build and then up in your pipeline template you could just say ci and that's going to kick off these three steps and that's really just for to keep the templates as easy to read as possible.

A

Another primitive that exists is keywords, so keywords are a way to inject variables into your pipeline templates. So a common use case that I use keywords for is to define uh regular expressions for branch names. If you need to do some branch, name, validation or conditional logic based on branch names, you can also use keywords for defining defining globals. So here we could just define that there's going to be a keyword called banana, that's equal to 11. and we could define a global's block.

A

That's going to have variables a let's call that 12 b is true and c is a b c. So now, there's going to be a variable accessible to the pipeline, template called globals and banana. So I can say println banana because it's it's just going to inject the variable name and I can say println globals.a.

A

Globals.B and globals.c, and let's just show that that is in fact, going to be accessible so here at the bottom, we've got uh the value that we had for banana the value that we had for globals.a dot, b and dot c.

A

So these these uh template primitives are there to make your templates as easy to read and to write as as possible. So let's go over one more feature that jte has to offer today and that's that's called. We call them life cycle hooks.

A

So let's talk about a use case where let's say I wanted to send slack notifications, um you wouldn't want to hard code, your slack logic into every other library that you have. You know that kind of defeats, the purpose of being able to plug and play with these different steps that exist so jt's approach to solving that is, is what we call life cycle hooks. So, let's create a slack library.

A

You know we are not going to update the template at all, but we're going to see a way that we can add slack notifications. You know dummy slack notifications at different phases of the pipeline automatically, so we can save this configuration.

A

We can go back to vs code, we can create a slack library and we can create a step called uh hooks dot groovy and this file can be called anything you want it to because we're not actually invoking it directly from the template. Instead, what we're going to do is utilize, these special annotations that jte has let me create them and then I'll explain them real, quick.

A

So there's a couple different options that are available, there's init, which runs at the beginning of your pipeline. There's before step and after step, there's uh cleanup uh and there's notify.

A

So, let's finish creating these methods at notify and we can add some print statements to all of them: uh print ln in it and, let's add one for before.

A

Let's add one for after, let's add one for cleanup and let's add one for notify.

A

And then, let's run the pipeline again, let's head back to our browser and just to just to refresh didn't add anything inside the pipeline template. All we had to do was load the slack library which has these hooks and when we run the pipeline.

A

Oh, so that's a fun bug. Let's run back, so you can't actually implement a method called notify. It's a reserved method. So if we look at the error that we got you're not allowed to override the final message notify on the object class, so that's easy enough to fix, we can say notify hook and now that will work as expected.

A

So we rerun this job again and we'll see that these messages from our hooks are getting printed out at a lot of key places. So again, jte will tell you uh what it's doing as it does it right? uh You know one of the honest drawbacks to jte. Is you have code coming from a lot of different places so to facilitate troubleshooting and debugging?

A

These messages make that really helpful to figure out.

A

Where is this output coming from so here we can see that there's an init annotation, that's going to run the init method on the hook, step from the slack library and it just prints out the message and now for each step that we run. We've got the before step the step itself and then the after step, and then the notify hook so visualizing. This can get kind of tricky. So let's go take a look at the jte documentation and under library development. We can look at life cycle hooks.

A

Let me zoom in a bit to make this easier to read. So here's a picture, that's sort of my face is blocking it. Here's a picture that sort of shows the way that hooks run as part of the pipeline. So in the beginning, you've got a validate annotation and an annotation for every single step that gets executed.

A

You've got a before step and after step by notify and then at the end you have clean up and notify again. Sometimes you might not want to execute every hook before and after every single step. So if we look at the documentation, there's something called conditional execution where you can pass a closure to these annotations and the return value of the closure is going to be used to determine whether or not the hook should be executed.

A

So we're coming up on time, but life cycle hooks are one of my personal favorite features that I love to share. They're, really helpful for things like notifications for sending pipeline metrics to you know something like splunk or elasticsearch, so that you can visualize them in grafana. Hooks come up in all kinds of fun ways, so the library development documentation will actually show you all the features that jte has to offer. So when you parameterize libraries, there's functionality around library configuration files they'll, let you do some type validation on your library parameters.

A

Here's an example of how I would recommend documenting these libraries so appuzalen. We have pipeline libraries that exist every single library that we have has a readme that allows you to see.

A

Let's pick on senor cube allows you to see what are the the library steps that are contributed when you load the library and what are all of the uh configuration options for that library. There's. Also a newish repository uh called jt library scaffold that sort of helps you get off on the right foot when it comes to starting your your libraries, so this repository integrates with release management. We use release drafter for release, notes, unit test for the libraries through jenkins, spock npm groovy lint for linting.

A

It updates the library structure a bit so that your every single library that you have can have a readme file. So within each library you create a test directory for your unit tests. You can add a readme for the library documentation.

A

Here's an example of that library structure and this repository is a way to automatically integrate those library readmes into a nice user facing uh library, documentation site um so that you can communicate with your users. What are the the libraries available and the steps that they can contribute? So you can build sort of a pipeline as a service framework so that folks can configure their libraries using your building blocks instead of having to write them all from scratch, so feel free to check this out feel free to.

A

Let me know if there's any feedback, I'd love to talk about the different use cases. So, to summarize, the jenkins sampling engine is a pipeline framework. It allows you to create tool agnostic pipeline templates that can be reused across as many teams as you want simultaneously.

A

You know you no longer have to create a new jenkins file for every source code repository. Instead, you can define that common template and then share common configurations, but also allow teams to specify what's unique to them right. So the the takeaway here is that it's a simpler way to manage large-scale devsecops pipeline adoption across the enterprise.

A

While having enhanced pipeline governance right, you don't have to go double check that every pipeline is implementing what you thought it should.

A

So these templates really allow you to define as an organization what are the required steps of your software delivery process and then give you a mechanism to implement that, while still giving teams the flexibility they need to choose the right tools for the job so feel free to join the jte gitter channel uh we'd love to hear from you feel free to star the jte repo and open an issue uh we love to hear from from our end users. Thank you for for taking the time to talk today. Bye.