Continuous Delivery Foundation cdCon 2021, 6 Jul 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Adding Ortelius for Microservice Tracking to your Jenkins Pipeline - Steve Taylor, DeployHub

Description

Adding Ortelius for Microservice Tracking to your Jenkins Pipeline - Steve Taylor, DeployHub

As we move into microservices, we move away from a traditional check-out and build continuous integration process where an application binary is created and versioned. We are faced instead with managing independently deployed microservices across the dev, test and prod states, where the ‘link’ so to speak is done via APIs and runtime. In this session, we will cover how to include Ortelius for managing microservices in your Jenkins pipeline. We will cover the maintenance of a ‘logical application’ view and version, bill of material reporting, difference reports and a ‘blast radius’ report showing the impact a single microservice update could make to any single cluster. If you are organization is using Jenkins, this presentation will give you some ideas in how to easily pivot Jenkins to support a Cloud Native Environment.

For more Continuous Delivery Foundation content, check out our blog: https://cd.foundation/blog/

A

Hello, everybody, my name is steve taylor and welcome to adding microservice tracking to jenkins with ortelius.

A

So one of the things we do is we love jenkins. We know that jenkins can do a lot for us and we want to see how we can uh evolve jenkins to work with hundreds of microservices we've had a significant increase in microservices.

A

You know six to eight months ago it was 30 micro services. Companies had and now we're into the hundreds 200s, and uh we foresee a lot of people approaching the the thousand mark here very quickly.

A

So my just to give you a little introduction about myself uh again steve taylor, uh cloud native evangelist and uh passionate about uh microservices.

A

uh I am the cto of deploy hub and the co-founder the community director for the artelias open source project, which is under the cd foundation and I'm the creator of openmake meister a build management solution that allows me to have the devops devops experience uh for over 20 years. uh If you want to chat, just go ahead and reach out to me on linkedin there's my linkedin address.

A

So let's take a look at where we are today.

A

um So today we typically have monolithic applications and those monolithic applications are configured statically during the compile and link process, and this all happens way early in the uh the build process I mean in the pipeline process, so this is happening at the ci level, so we have the developer, checking in the code and from there it gets compiled and linked, and we create this monolithic application that has everything resolved at that point.

A

uh Typically, there's a build manager or build meister that is responsible for the scripts that are going to be used to create that monolith application. You know these are in the form of maven scripts gradle and make scripts.

A

You know meisters build configuration files. Any of that is used to pull together the pieces that we need for creating our monolith application. What ends up happening in that process? Is we end up interacting with our version control?

A

Like git, we pull our code down, we'll interact with artifact repositories if we're building a java application, we'll be interacting with like nexus or artifactory, some sort of maven repository to pull and push artifacts to and what ends up happening in this process is we have our big build that creates our our application binary, and we know that it's a specific version.

A

We've pulled together these pieces and of source code and libraries and we've stuck them all together and packaged them into our basically our application executable and that's going to get versioned into a repository uh from this. We can actually see a bill of material report. Actually what went into uh the the build at that time, so we can get the the dependencies also. We can compare uh what happened from the previous build.

A

You know we are so close to the the get repo at this point um that we can actually see coding changes between by comparing commits at that level. So it makes it very easy for us to actually have a version of of our application.

A

We can move that version around it's a usually a single file or a couple files. We can see what's in it and we can see what's changed now when we look at microservices. You know one of the goals of microservices is that they're built and deployed independently.

A

You know they're they're supposed to be loosely coupled, uh and this means that they can have their own path uh through the life cycle. A microservice developer will typically uh be working with some flavor of a container.

A

It may be serverless like a lambda function or it may be a python flask, that's being put into a docker container, that's going to run in kubernetes.

A

Now what ends up happening is when we uh start thinking about a microservices. These are very small transactional level pieces of code, so the goal behind a microservice is to handle only one specific uh set of functionality. So if I want to get the user profile, that's the only thing that's going to do. It's not going to deal with login or you know, credit card payments um like a monolith does where monolith has to do everything under the sun.

A

The microservice is going to have one specific function that it's going to do and because of this, we're going to have lots of them. We're gonna have lots of different independent microservices.

A

They may talk to each other um and we may have dependencies between one version of microservice to another, but the whole goal and is to make them backer compatible and independently deployable.

A

But as we look at what happens when we go from the dev world to testing, we actually keep on duplicating or redeploying uh existing images out to these different environments. So if we have uh 30 microservices that our application is made up of- and we have four stages to our pipeline, we could potentially, depending on how we architect things end up with a 120 different microservices that we need to keep track of. We know that there's going to be duplication between all the different stages of the pipeline. We're the same.

A

You know we don't change every single microservice for every single deployment. So in this, in the microservices uh realm, we are doing a loosely coupled um type of architecture where we actually are actually linking at runtime uh in the monolith, we're doing the link and resolving all of our dependencies at build time here with microservices, we are actually resolving those uh that link step at uh runtime.

A

So this is, after the release has been completed in the monolith. They we resolve it before a release is done so, what's this uh what what happens when we deal with this large number of microservices?

A

We find that we don't have a clear understanding of what clusters have a microservice on it or what which namespaces need to be updated.

A

We it's very hard if you're have a you know a good number of microservices to figure out who's consuming mine. Like I talked about, we have the possibility of a service to service relationship or a service to the consumer side. We need to know who's using it. So if I need to change it and cause a breaking change, I have to know my blast radius. Who am I going to impact, or if I have a bug you know, and I need to let other teams know that there's this bug that we're going to fix?

A

Who do I need to update, so it's really important to find out who's consuming my service, then also, you know once we roll it a service out. How are we going to test it? What is the logical application that we need to test? um Are we going to just test to do a string test on the service itself and say that is good, or do we want to tie it together with all the other services in, like the the consuming front, end piece and do a full uh test as part of that process?

A

The one thing that we don't really want to do is wait and deploy uh wait for an instant incidents after we deploy you know, so we want to be proactive and not reactionary as part of that process. In order to get there, we need to reimagine our process.

A

So how do we do that? We can think about our continuous integration process uh and the build step a little bit differently. We have to think logically, instead of trying to piece it all together and do a monolithic type of style of deployments and that type of processing, we really need to think about how we can incrementally deploy and pull back the logical view of our application.

A

So that's where we came up with the uh artillius open source project or to this is a microservices management platform. That versions tracks microservices figures out what the blast radius is uh we'll provide an inventory across all the clusters and be proactive in seeing how your microservice architecture is changing over time. um Ortilius, like I said, is part of the cd foundation and it's a incubating project in this. In that uh governance, our mission is to simplify adoption of microservices through a world-class management platform driven by supportive and diverse global open source community.

A

So in the open source community for ortelius, we have contributors from all over the world, all different skill levels and, if you're interested please come join us. So how does ortus help evolve? The jenkins pipeline?

A

Well, one of the things that we need to look at with microservices and jenkins is we have to look at uh the data that we want to get out of jenkins and how we can bring that back to give us that logical view of an application.

A

So whenever we build a microservice we're going to hook into the jenkins pipeline and we're going to track that that microservice has been created and also we're going to look at the relationships that that service has to not only other services but also to the application level. This allows us to do a level of build material reporting.

A

We can do that at the individual microservice level or we can roll up to the the logical application view we can produce different supports, so we can see what's changed between one version of the service to another version, but better. Yet we can see what's changed between a version of your application and another one.

A

Like I said earlier, when we get into you know hundreds of microservices we're actually going to find that a small subset are going to be the the frequent uh changers, maybe 15 20 after a while, we'll have 80 percent of that code base and those microservices are going to be pretty static. You know we may get some bug fixes along the way, but um those would become pretty solid and because of that, we can incrementally deploy our application and just deal with the change.

A

So difference reports is very interesting in that area and also the blast radius. Like I said earlier, who's consuming my service and how am I going to affect them and then who owns the service?

A

How do I get in touch with them? What is the slack channel or the discord uh channel? What's the project owner's email address? uh You know that type of information.

A

So if we have problems with it and we need to dive into uh the the hood, we know who to talk to at that level, then also uh through the integration between ortulius and jenkins, we're able to bring back that logical view of the application, and this will automatically allow us to do a level of versioning that we lost when we stopped publishing a monolith to a repository.

A

So ortillius will bring that back to the picture for us around microservices.

A

So it's really reimagining uh how we can use jenkins and leverage its power with ortelius to give us the the information that we need to be successful.

A

So where does artillerius fit into our pipeline? So one of the things that ends up happening in our jenkins pipeline is we'll actually end up doing a docker, build and, as part of the docker build.

A

We end up pushing after that's been completed. We'll push that result up to a docker registry that that, once that's complete uh ortilius is going to step in here and gather information about uh that service. That was just created. um Some of the things that we're going to track is going to be like the git commit uh the branch, the repo.

A

Also, where was this image pushed to um which container registry? um What was the tag? What was the sha? Those type of things is what we need to know at this level of the pipeline.

A

The next you know, depending on how we set up our pipeline. If we're going to do automatic deployment or if we have a gated pipeline, doesn't really matter to artelias. But what will happen is we will need to do a deployment?

A

The deployment can be done by ortulius, it has a deployment engine built in or you can use uh another deployment tool and ortulius record what that other deployment tool is done, but in the end, what we want to do is is track where this component, that we just created or this uh this microservice that we just created, where did it go to, um and that gives us that map uh to let us know how things are put together and where they are, are living in in the runtime environment.

A

So what we do on the ortus side is we use our restful apis to interact with uh the jenkins pipeline and we store this metadata that we're collecting this catalog data around the microservice and pushing it up into the orotellius postgres database.

A

So, if you're interested in getting started with ortulius, you can use the deploy hub, free sas version or you could do the online version. There's a couple links for you to look at on how to install on-prem or sign up for the the team version there.

A

You can also reach out to our discord channel to get any information any help on that process. So how do we actually get this done inside the jenkins? So uh there's a couple ways. The way that we're gonna go over today is the ortilius groovy library um that's out in our git repo.

A

um Also we do have a python library as well as a command line and a docker image, so in this case we're going to go over uh the groovy library because it gives us the most flexibility.

A

So here we're adding in uh the uh the get repo for where that library is going to come from and we're just going to pull off the master branch there. So what this is going to allow us to do is load in the ortelius library.

A

Here's a sample jenkins file, one of the things that we're setting up here is, after we clone our code from the github, we're actually going to track information about, what's being built in this case. We're setting up some parameters for how to connect up to our artelias instance and we're defining what we call a component.

A

A component in arterius is the same thing as microservice.

A

um You can have components of of many types uh in this case we're going to be talking about docker components uh in artelias, so we set up our our component name, our application name, that this component is going to be consumed by uh what our version of our application is, uh what the image name is gonna, be when we push it up to our repository and we set a couple, a few other parameters as part of this process. So basically, this is a lot of this information.

A

We can gather from your jenkins environment, you know if you have a standardization on like the uh project name: uh workflow name, you know um global environment variables. All that information can be uh pulled together from your jenkins world. So our next step is, we actually have to uh create our component.

A

So after the build's been completed and we've pushed to the docker registry, we want to go ahead and create a new version of our our microservice, our component, and this is where we're going to pull together uh attributes from the jenkins environment like in this example. Our build number uh is being pulled from the the jenkins build id number and then uh other things. You know the chart version that type of thing uh just some attributes that we're going to set at that level.

A

So what this allows us to do is we will actually create a new version of the component. uh If one doesn't exist, it will go ahead and create the base one and if, when one does exist, we'll take the previous one and use that as our starting point and keep on iterating up the uh version chain which, uh with the new definitions so after we do the component creation, we're actually going to take those adder attributes and assign them to that new component version.

A

So this is where we're making that link between what jenkins and docker have and storing that inside of the ortillius database, which allows us to then down the road when we do deployments reference that information and then our next step after we update our component attributes, is we're going to create a new application again. This will go ahead and look for a previous application version. um If there's one there we'll go ahead and use that and then we'll incrementally apply this new version of the component to that application.

A

So this allows us to do that incremental uh updating of our logical view of our application from jenkins. So it makes it very easy to self-maintain your versions of your application and your components uh through the the date or to this groovy library.

A

So next we're going to go ahead and add a few attributes to the application so attributes you can think of as key value, pairs or config map data can be assigned to uh components, applications, uh environments uh or endpoints, and that's what we're doing here. We're assigning a couple more parameters to the environment as well as the end points.

A

So this is giving us all that configuration data stored inside of a central repository in the in the ortilius database, which we can reference we can get uh do reporting upon. We can do that. Those questions that we asked earlier. What is you know? What's my bill of material? What are my differences? uh Those type of things impact analysis, those types of questions that we need to answer all that data. All those relationships will be inside the uh of artelias.

A

So this is kind of a look at what happened. As we went through our build process for a component, we can see that the jenkins job that actually kicked it off the build number the build url. uh You can see the container registry, the digest the tags. uh What was the git commit that triggered this whole process?

A

Moving along all this information, we're going to be gathering from the jenkins pipeline, we can also store additional information at the component level or at the application level around. You know, service catalog information, like I said, who's the owner. What's the um the slack channel? What's the pager duty service url, you know what are the cve licenses that are being consumed by the service? You know what are making up?

A

What are the underlying uh pieces of the service, so all that metadata all that service catalog information is rolled into uh ortulius for your use later on, when you need to get to that type of information quickly.

A

So our next step is that we ended up doing when we assign the component to the application version is where we actually updated this uh this map. So this map lays out all the dependencies that are in. In our logical view of our application and as part of our ci cicd integration to jenkins will self maintain this so we'll find the previous version of a service and we're making an update to it. So we need to know. We know that we need to uh replace that one as part of that process.

A

So we can see that uh we'll end up with a a lot of microservices, like I said and they'll have different uh locations that they can be stored in uh in arterius. We can organize our services into domains around a domain driven design.

A

So here we can see that the one that's highlighted is in the cart service domain, the one right below it is in the checkout domain.

A

So this gives us a way to logically group together microservices, regardless of the jenkins organization and the workflows there, because the workflows are pretty flat. We don't get a lot of organization, we can put some into it, but if we want to get crossover business silos, ortelius can do that for you through the domain-driven design. So we can have one team pushing to cart service domain and another team.

A

That's using a totally different, it could be even totally different jenkins instance, pushing to the same cart service for their uh cart service, mic cart, service type of microservice, so this allows us that that smart tracking of what's happening at the microservice version level so from there we'll actually roll up uh and package together all of the services that we want in a version of the application.

A

So in this case um we have a couple versions of our hipster store, labor day sale and you can see the the latest. One is the 10.0 version. Now this one was built, it was built based on the 9.1 version, so the only thing that's going to be represented. There is going to be the differences, so it makes it very easy to get a nice view of what's changing from version to version of our application.

A

Even though everything's spread out, um we have micros, you know hundreds of microservices across multiple jenkins pipelines, um multiple jenkins instances pop, possibly we're going to pull all that together with artelias and give you give that logical view back of what your application looks like, and what this allows us to do when we start looking at an application is give us our bomb report. So we have two versions of our webstore there and we can see where things are being shared and where things are um new versions uh being consumed.

A

So, if you see a orange box there with two arrows, that's actually a shared service between the two different versions of the application, the ones that have a single arrow. Those are independent, like new versions of a service that the new version of the application is consuming.

A

So that really gives a gives us a look at how our application is being put together, what we're consuming if we go over to the right, where the blast radius report is. This is where we're looking at. From the other viewpoint, I have a service who am I going to affect so in this case our cart service is affecting three different application versions and that's important to know if we need to fix a bug or if we have to break backward compatibility upon it.

A

We know who we need to talk to at that level, and this will span across business silos as well. You know, there's not a orterius is designed for collaboration and allow you that visibility across project teams and finally, at the bottom, we have the difference report like I said, artelias can do deployments and what it does it can do them in an incremental fashion. In this case, when we deployed our new version of our application, we just ended up deploying the new version of the cart service, all the other microservices out there were the same.

A

What that means is our deployments are much easier. We understand the impact of if something breaks, what changed we can actually say. Oh, we just deployed the 150 version of the card service. Let's roll that back and isolate that to be the one that's causing the problem. So the difference reports allow us to do that type of uh insight into uh what's happening at the environment level, so that was a quick rundown of ortelius and how we fit into jenkins.

A

If you have any questions, please contact us, you can get in touch with me on linkedin or you can go out to the ortus website. Ortilius.Io io you'll see the how to get on our discord channel.

A

The main repo for artelias is ortilia. Slash, ortiz and github. Follow us on linkedin, and if you want to follow me on twitter um or on linkedin, there's my handles there sptailer15.

A

So thank you very much and reach out. If you have any questions hi there everybody. So if you have any questions, looks like they're giving us like one minute uh to answer any questions. I've been answering as we've been going along also to let you know that there is a uh another demo that we're going to be doing at uh this afternoon in our booth. I believe 4 15 mountain time so stop on by and it's in, between the break uh at the top part of the day. So we'll talk to you, then.