Cloud Native Computing Foundation GitOpsCon North America 2021, 30 Oct 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: GitOps and the CD Pipeline - Friend or Foe? - Tracy Ragan, DeployHub Inc.

Description

Don’t miss out! Join us at our next event: KubeCon + CloudNativeCon Europe 2022 in Valencia, Spain from May 17-20. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

GitOps and the CD Pipeline - Friend or Foe? - Tracy Ragan, DeployHub Inc.

While GitOps is solving some big problems for continuous deployments, does the operations by pull request model fit into a continuous delivery pipeline? In this session we will explore the ‘front end’ of GitOps and how the CD pipeline will need to shift to embrace a GitOps approach, particularly around independently deployed microservices. The question to answer is how will the CD Pipeline automate the GitOps steps that are currently done by a human, such as updating .yaml files, submitting pull requests and approving commits. With potentially thousands of .yaml files, automation of these steps will be critical.

A

Hello, everybody welcome to get ups con, north america 2021. I'm super happy to be here.

A

I am doing a 10 minute lightning talk today on get ops and the cd pipeline, so it's going to be quick and in that 10 minutes, my goal is to get you thinking about what the cd pipeline is going to look like when we have when we're using get ops to do deployments and we're in a microservices environment, things are just being disrupted by microservices.

A

The cd pipeline is particularly vulnerable to being disrupted because of microservices and gitops changes. The way we do deployments. So how are we going to do it with lots of microservices.

A

So, let's start at the beginning, get ops came from iac and you know infrastructure as code that worked really well, so sign up kind of morphed into supporting applications, and that's what we call get ops today and when we first started doing this, we did it for containerized applications, which meant that we had pretty much one yaml file. We created a yaml file for doing our deployment of our containerized application.

A

We created a separate environment repository to store the yaml file and the get ops operator basically reads from it, and that takes care of the problem for us and it's beautiful. It's audited. Everything works quite well.

A

But that was one yaml file per containerized application right, easy peasy.

A

When we did that, we had at probably maybe three four different ammo files we had to manage through the get ops process. You know in a typical pipeline, we have dev test and prod and the way we normally do this get ops process. Is we branch, the environment repository for that yaml file to handle the different configurations needed for dev test and prod? So when a new container gets a new application container got created, we grabbed the new shaw.

A

We updated the yaml file to address any environment, variable differences or key value pairs between the the dev test and prod environments, and we manage those yaml that single gamma file in those three different branches that allowed us to create a life cycle within that get ops model.

A

So and in some cases you may have just one main trunk with multiple yama files that the operator interrogates that's another way that could that it could be done, but most people branch it out.

A

So now, let's add this microservices puzzle to the to this particular scenario. So you know microservices can be complicated. They shouldn't be we're getting better at doing them. More tooling is coming out to help with it, but talking about microservices in a githubs model means that we're going to manage a lot more files and we're going to have to manage those additional files across the pipeline and those files add up kind of fast. um It's it's not. You know.

A

We talk to people who are saying they're, doing 10 micro services to 100 microservices to 500 microservices, and if we just go with 10 we're going to add up quite quickly, but in our scenario we're only going to have a few.

A

So let's talk about this we're going to have more than one gamma file in this scenario, where we have a candy store cluster and a hipster store cluster, we share between those shipping payment and the card service between those two websites.

A

Now, in this case, we're going to have four yama piles, yaml files per um per state or per cluster, we're going to have the front end of the candy store or the hipster store, and we're going to have the cart service, the shipping and the payment yaml files.

A

Now, if we um think about what that looks like going to a single one, big cluster, if the in this particular scenario, they have created name spaces for the cart, shipping and payment service, plus the candy store and the front end and the hipster store front end, and we have these yama files that are going out to support those. So it gets big quick.

A

So, instead of having one contain one yama file to support our containerized application, we're going to have lots of yammer files and that's what creates the the complexity in in microservice in many ways, not just on this one, but in many ways.

A

So now we're going to add the pipeline, we're going to have that environment for development, we're going to have that environment for test and we're going to have that environment for production. So it can add up a lot faster than you realize. So in this case, we have one two three, four five yama files, including with each of their front ends, and now we have 15 yama files that we're managing and that's all in different repositories, because each of those are going to be managed in their own github repository that represents their environment.

A

If you're doing it that way, other ways to do it, but still you're going to have a lot of yaml files, no matter how you slice, it.

A

So, as I said, we're getting better at really thinking about how to manage microservices. One of those ways is the emergence of the microservice catalog.

A

If you haven't heard of one before they're new, you take a you're really thinking about a service catalog, but it's a microservice service catalog and that service catalog is a unified catalog of microservice configuration data, including things like the shaw and key value pairs and and pretty important ownership.

A

It stores and versions that information and tracks the tag for each new version of a microservice, and this information can be used to generate yaml files based on the environment, key value pairs. So one of these problems is to take the human element out of the get ops process and start doing more code generation and a microservice catalog stores a lot of that. It hoards a lot of that information about a microservice and can be used to do that.

A

So it can be used to generate the ammo files based on the environment and it can be triggered. It can be a triggered event that creates a pull request in place of a deployment step, gets it into the correct environment with the correct data, the correct shaw, the correct key value information and can also automate the commit based on policies.

A

So you might be able to define policies that says this is good to go and you can automate the commit if, if the team allows it in essence, what I'm talking about is automating the the human side of devops all the way through the get-ups process, we've always been automating devops and we strive to in the continuous delivery kind of philosophy, and what we need to do now is think about how we can do that with a get ups deployment model, with lots of microservices.

A

So this new market is starting to emerge when it when we talk about service catalog. This is a great article. If you don't follow tyler jewell, I highly recommend you do.

A

He writes some amazing well-researched articles he recently in this article he he forecasted two trends, one the emergence of the cervix catalog and two, the convergence of continuous delivery and get ops and the new market that that will create, and then actually, if you, when, you really look at the problem set both of these these markets, the service catalog market, around microservices, along with this convergence of get ups and continuous delivery, is what's really starting to create this. This new marketplace.

A

So think about the pipeline as looking like this, you get a container registry push. So as soon as the a new container has been registered to the to a container registry, something gets triggered to update the catalog and when it does so, it pulls all that shaw information.

A

The catalog then can get records that data until a deployment is triggered at the point in time that a deployment is triggered, then the the yama file is generated and a pull request was created once that commit is executed. We're going to put that we're going to push that that forward and it's going to be pushed out to the the end targets. Automating. These steps are what will be critical to be able to achieve success in both microservices and a get in a get ops model and oh yeah meet the speaker. I didn't know.

A

If I'd have time, I am tracy reagan. I am the ceo of deploy hub, I'm a microservice evangelist super passionate about configuration, management and continuous deployments. I used to be a founder founding member of the eclipse foundation and now I am a board member of the cd foundation. Who's really really pushing these conversations.

A

You can follow me on twitter, at tracy reagan or at my linkedin at tracy reagan dash oms, and I encourage you to learn more about catalogs. The linux foundation has two open source projects. One is artelias through the incubating through the cd foundation and backstage, which is a sandbox box project through the cloud native computing foundation.

A

Thank you so much for listening. I hope I have stimulated some thought and let's solve this problem together. Thanks.