Cloud Native Computing Foundation GitOpsCon EU 2022, 19 May 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Lightning Talk: GitOps and Progressive Delivery with Flagger, Istio and Flux - Marco Amador, Anova

Description

Lightning Talk: GitOps and Progressive Delivery with Flagger, Istio and Flux - Marco Amador, Anova

Organizations that use progressive delivery are able to ship new code faster, reduce risk, and continuously improve customer experience. Progressive delivery is an essential component of DevOps, and feature management is the primary way it works. In this talk, Marco Amador (Anova) will describe their journey into progressive delivery with some hands-on demos and explain why they've chosen progressive delivery on their multi-cluster and multi-region Kubernetes cluster.

A

Okay, hello, everyone, I'm mark commadore software, lead for data engineering and sre at inova, and I'm here to present a little bit about a novice journey on githubs and progress delivery.

A

So anova is the global leading provider of industrial iot and is the result of a merger of five well-established companies, and with that we had five platforms and five different ways to build software and deliver it. So we created a new platform trying to follow the cloud native culture and best practices and created the unifi platform.

A

The platform that we've created as a state-of-the-art tech stack the cloud-based tech stack. We are running everything on kubernetes on these two service mesh and with a pretty nice observable stack like prometheus and and loki, and so on.

A

We had a few key aspects to take care when creating a new platform from scratch. One of those was scalability.

A

We needed our clusters and workloads to scale automatically accordingly to the load and we needed to create a multi-geography and the multi-region clusters, because we need to segregate the data from different geographies from our customers and multi-region basically for data replication and redundancy.

A

We have multiple environment staging and production in all of those geographies and, of course, we needed to enforce our security policies in all of these clusters.

A

More importantly, we have different product teams that we wanted to have completely full ownership on their infrastructure and applications.

A

So we also need to a way to configure the roles and the tenants themselves, and for that we used gear ups, more specifically flex. We started with flex v1 and for a while now we are using flex v2, which brings us a few controllers source controller to handle all our git repositories, customize controller. That applies customize on those git sources and create the the kubernetes resource themselves and elm controller to take care of the m releases and perform all the elm operations that we need to install our elm releases.

A

As I said, we need multiple git repositories, one of them to our platform team, basically, where we describe all of our elm releases for operations related like the observability stack, the hd service mesh, the tenants themselves we need to specify here and the permissions and each tenant has their own git repositories where they describe their own workloads at will.

A

So with so many geographies and environments and visions, we needed a way to keep our repositories without duplicated code and something to prevent drift between so many environments.

A

For that, as I said, we are using elm releases from flex is a crd and customize that allow us to apply overlays on top of some base or common base manifests, and then we can apply the small difference that we have between environments while keeping the base.

A

The base clean and without duplicated code yeah, I usually show my wrappers. I have some example repositories on my github that we can check after this. Well, how do we continuously deliver our workloads?

A

We use a pretty nice picture of flux as well, which is the image automation that brings two additional controllers: one, the image reflector that keeps fetching the new image stacks, that we are keeping pushing in our ci pipelines and another one that matches or try to matches our image policies with those image tags that we are finding on our container registries.

A

If any of those match the image policy. With the new tag, a new patch is applied in our or in any git repo that reference that image policy, and then we roll out the new elm release with the new version, but not so fast, because sometimes we don't want to simply roll out the new image, because a lot of things or a lot of bad things can happen. We might be introducing a regression.

A

We might be missing some integration tests that we forgot to to do, or even suddenly we might, our apis might start taking longer that our slo allows. So we are using flagger to that as many deployment strategies, scanners, a b testing and blue and green. We are mostly using canaries where the principle is very simple: we start, or we spin up new containers using the new image that we detected in our image policy and start progressively sending or forwarding traffic to those containers, and during that which is called the canary analysis.

A

We we keep monitoring the metrics or whatever we want to see if the slo still complies. If everything is okay, the canary is promoted to the primary workloads.

A

Yeah here is the kind of the chart that explains this a little better. We start progressingly.

A

Forwarding traffic to the canaries if everything goes well, keep increasing that amount of traffic and if everything is okay at the end, the the primary release starts using the new version and the countries are killed.

A

uh Sometimes we don't have enough traffic to have metrics, so the categories wouldn't progress without the metrics for us to analyze. So we can use different strategies like using generate our own traffic, like using tools like a or k6, where we can perform loading tests uh during the canary analysis, uh while looking and seeing how the countries are progressing, we can keep seeing this in our dashboards to see how the canaries are performing.

A

um The canaries are ephemeral, we test it and the canal is only last while we are doing the canary analysis. Sometimes we want to keep the new version not being exposed to all of our users, so we are using dark launches where, for the case that we cannot test the new feature using feature flex.

A

For instance, when we start using a completely new architecture, we wanted to make available the new, completely new version to only a specific set of users, and for that we are using um an issue virtual service fixture, which is called delegation where, depending on an http editor, for instance, we can delegate the traffic to another virtual service from from istio. In this case, if anyone knows the other side dark, we'll hit the new versions that I'm releasing and are available only for the users that know that specific error we can keep running canneries the same.

A

We have all of this set up in our elm charts and different to the canaries. When the canary analysts finish, we still have the new version available for the ones that know that specific header and basically it's what we have been doing in with gear ups and progressive delivery.

A

We still have a few challenges like we needed to. We have a different observability stack in on every um cluster we'd like to have a unified, for instance, observability stack and that's it. I hope you enjoy it.

B

We do have time for questions if you want to take a question. Just go to the microphone. If you'd like to do a question, there's a microphone in the middle. If somebody's up front, you can wave your arms and I'll throw a mic at you. You have four minutes any questions.

B

We got one I'll bring the mic over.

C

Thanks for the presentation, it was very good and very interesting part, so I liked your choice of the stack just one question. You mentioned that for the canary release, you don't have sometimes real data, so you execute tests, for example.

C

Is the quality enough to be sure that it's good enough to release it because you're creating tests on your own? Maybe it's not really the behavior of the users you trying to see.

A

True true, but we are using uh specific containers for that specific service where we define a bunch of case tests that we perform, I think, probably, is even more accurate because we can just go through.

C

A

Entry point that we have to test it while, if you are only getting traffic from outside, we might be eating only a few of those services, and then we are not really analyzing the category correctly.

B

Any other questions raise your hand or go to the mic in the middle of the room.

B

We have a taker, raise your hand if you, if you're gonna, have a question as well and I'll get you set up next or line up, go ahead.

D

I was just wondering how are you keeping istio and flux up to date in your production clusters.

A

uh Well, we are, we are keeping upgraded uh with basically manual. We are right now we are using h213, I guess it's pretty updated and the flex is the only thing that we actually are not using image automation, but we could also use image automation to to update flex itself, but we are not doing that. We are actually performing because we want to test first in staging, we are doing flex install and so on, but flex it's in its own github repository.

A

So we are doing everything as good, but we are updating that our operations github skit manually with flex cli itself yeah. Thank you.

E

C

Really enjoyed.

E

The talk, um how are you securing your get repositories to to make it so that only kind of privileged people could you know.

A

E

Code to production.

A

Yeah, basically we have we are. We have everything on other, so we have id groups that only the members of a specific tenant only have access to that git ops repository that they own.

B

Thank you yeah another one.

F

Yeah, thank you for the presentation. So how do the application team and the developer uh sort of interface with flux? Do they decide the threshold for a b testing and.

A

How is that configured yeah? We have one single album shark, that supports it's very flexible, so they can decide what latency they. Basically, they can decide. There's slo for that specific service that can be latency can be error rate can be whatever they want. They have complete ownership. The amstarts are flexible enough to be configurable as we want.

B

All right, thank you over everybody, give a round of applause. Thank you.