Cloud Native Computing Foundation KubeCon + CloudNativeCon North America 2022, 11 Nov 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Hands-Off Features Releases With Keptn, OpenFeature, And Open... - Michael Beemer & Johannes Bräuer

Description

Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe in Amsterdam, The Netherlands from April 17-21, 2023. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Hands-Off Features Releases With Keptn, OpenFeature, And OpenTelemetry - Michael Beemer & Johannes Bräuer, Dynatrace

Speakers: Johannes Bräuer, Michael Beemer
Releasing a new feature into production always comes with an inherent risk of introducing issues. The code may have been thoroughly tested in lower environments but differences like environment specific configurations can cause unexpected behaviour. Feature flagging helps reduce this risk by allowing a subset of users to verify a new feature in production before exposing it to all. But how can we achieve hands-off feature release automation? This session will demonstrate how feature flagging and life-cycle orchestration work together to automate feature releases. Keptn will be the orchestration layer to automate feature validation. It works with OpenFeature to control access to the feature itself, allowing automated tests to verify a feature before it's generally available to users. Keptn will then react on the test results by either progressively enabling the feature for all users or initiating a troubleshooting workflow using OpenTelemetry.

A

We are super excited about our talk with we Amin Mike Beamer, who is maintain our open feature and myself Johannes poya, open source project.

A

Why we are excited, because today we want to bring together open feature Captain as well as open Telemetry, in order to set up a releasing environment that allows us to release features in production without or in a very safe manner.

A

For our talk, we have prepared a couple of sections. Mike is going on to explain why it's important to have feature flagging adults. Afterwards, he is presenting more details on open and then I will jump in and I will talk about Captain in the demo. We are then bringing together Captain open feature and open Telemetry to release a new feature in the hands-off manner.

A

And finally, we will then do a recap: all right, Mike, let's get.

B

Started all right thanks, Johannes yeah, so just a quick introduction to feature flag.

B

um Basically, it's a software technique that allows you to modify system Behavior Uh without requiring a redeployment, and so it's really powerful. It's used in a lot of ways very commonly it's used for like as a release toggle, so you can put new functionality behind a feature flag and then enable it when you're ready. You can use it for experimentation, so a b testing or ABN testing, where you would run experiment, collect data, analyze the results and then choose the best of the options.

B

uh You could also use it as an Ops toggle, and so that's another way to look at feature Flags. Typically, you would think of a feature flag is something that you would add your code. You would use it until it's. You know on or in a stable State and then remove it and Ops toggle would actually be used for basically the life of the project, and you would use that in situations where you need to disable some kind of behavior to maybe save your system, some load or if another third party service is down.

B

You can basically use that in operation mode and the last one is like a context aware: toggle, that's something that's going to be really important for this demo today and that's basically where you can pass in different contextual information like what users logged in what their Geo is or anything like that, and then you can make decisions based on that metadata.

B

uh One way to do that or one of the big benefits of feature uh uh is basically you're. Decoupling, your feature releases from a deployment. So if you think you're writing your software you're doing your deployment you're, sending it out to production traditionally, that's basically it you'd roll it out to production and all of the features are immediately available with feature Flags. You can basically do a more granular release, so we could enable a feature uh specifically or or actually release it.

B

That's already been disabled and then roll it out to a subset of users until you feel like it's safe. So that's what we're doing in this example here. So you imagine you're in production, we have one new future flag and then maybe we're enabling it maybe for a product manager and a synthetic test, but your customers are seeing something else that would allow you to test it out, make sure it has the intended behavior and then go ahead and roll it out safely.

B

So moving into open feature open feature is an open standard. Vendor neutral feature flagging specification. We have broad industry, support and there's a lot of interesting. You know thought leaders in the space that are also involved in this project.

B

uh We have sdks available in many popular languages right now we have java.net JavaScript and Slash typescript and go with many more on the way, and if your favorite language isn't supported, it's likely either coming soon or we would love your support. If you want to help build it, we also have a developer first Cloud native implementation. That's something we'll show it off in the demo as well, and it's just a way to test out the a lot of these feature.

B

Flagging Concepts, without having to you, know, register for a vendor account or build anything from scratch. uh On on your right, you can see kind of where open feature fits into this whole architecture. You know you have your application and in the yellow, you see the open feature, SDK and a very important concept here is the provider, and so that would be basically the the piece of code that would perform the flag evaluations, and that could be anything it could be a vendor.

B

It could be your homemade solution or it could be one of the open source offerings that we we provide as well and open picture was also just recently made a Sandbox project as part of the cncf.

B

See me ask why why develop a standard? So really one of the the main benefits is just a consistent API for developers. So, if you're getting started with a homemade solution, you don't have to. You know, come up with your own patterns here you can just leverage our system uh if you're going between different vendors, if you're you know changing from a homemade solution to maybe a commercial solution, the apis are the same.

B

Because of that we can actually leverage the community to help build really nice tooling around this, which is really really beneficial and, as the community keeps growing more and more interesting use, cases will come out. uh It also lowers the adoption risk. So, if you're getting started with feature flagging, you don't have this like code level lock-in that makes it really tricky to move between different providers uh and from like a dinosaur's perspective. We actually have the ability to do uh Telemetry data, so we can.

B

Actually, since we have a single SDK, we can actually capture interesting data points on flag evaluation and then provide that in Monitoring Solutions as well.

B

Just wanted to show you. uh You know it's not just like a open feature by itself. We're actually working with a lot of you know, companies that are pretty well known, some names you may recognize uh and we have a lot more on the way. So these are the ones that are available now.

B

uh You know just to highlight you know: Cloud V's, launch directly split many many others are available and then some open source projects like go feature flag and then flag d speaking of flag D, that's the one that we'll be using in the demo today and that's just a you know: spec compliant implementation. It's it's built. You know with a cloud in mind and kubernetes in mind and it does follow like a Unix philosophy. So it just does one thing and does it really well I'm going to hand it over to Johannes? Now all.

A

Right, let's talk about Captain as a project achieved incubating this or became an incubating project this year. This made us very proud and it's basically known as an application lifecycle orchestrator now you might ask yourself what can I now get from Captain and for digging into that? Let's first understand which problem kept.

A

And therefore, I want to jump back about three years ago, as we kicked off the project, we asked quite a lot of end users. What are actually your biggest troubles and challenges when it comes to Cloud native delivery and the effect that we got was basic, that they are constantly faced with the problems around keeping the pipelines up and running.

A

One example: I show you here, it's Christian who was our senior, therefore attention RIT. He told us that he has to deal with 2800 projects and there he needs 1000 Pipelines that he needs to keep up and running.

A

The numbers are high, they're, really bad or comes in.

A

When we take a look at the pipeline itself, because those pipelines they are quite heavy and quite long in terms of lines of code and the the engineers and developers they also baked in quite a lot of tooling, which made them really complicated to maintain and to run over time- and this yeah, you can imagine- escalated really quickly and became almost impossible to maintain, and this wasn't the time where we thought it might or there needs to be a solution which can deal with that um problem and in order to explain you what captain now solved for you, I'm, showing you here visualization of and that uh Christian was gave us.

A

Basically, the pipeline contained a bunch of tasks that should that's totally fair. As we wanted to information. We have to do certain uh tasks and steps. However, those tasks they are and were strongly connected to the underlying Tooling. In other words, they were a strong and hard dependency of what we want to execute and how we are going to execute that and here Captain jumped in, and we kind of uh proposed that we are now going to break apart. What the process is from what the underlying tooling is actually and the way Captain is working.

A

It basically introduced and right in between this Eventing layer allows the Captain sends out the events and then the tool is picking up those events and executing the tasks they are designed for, and this brings in now quite a lot of flexibility, because I can easily exchange the tool without touching the process or yeah working around within the pipeline.

A

Each band is based on the cloud event and some metadata, which is required to execute the task, but not really, and with that approach we are now able to have. On the one hand, the process definition separated from the actual tool tools that are then executing the process.

A

Now, with a well-defined process, you can then share this across your organization, and each team is following the same process, but using other tools for executing it, and you can also think of using this product stage. Experiment, whether it's the development to execute this process.

A

Ing environment, where you want to run the process, and even in production, you can also run this process. Foreign.

A

All right, I think this is better um now so important to understand is that Captain comes with two concepts: there's first, the concept of the shipyard, which allows you to declare how the environment should look like a shipyard is here shown in the Middle, where you can see that we have defined an environment with two stages: a development stage and a staging stage, and within each stage there is a delivery process which is which consists of a four tasks.

A

To execute important here is that in a shipyard configuration there is no um reference to a tool, because this is extracted and defined in what we call the uniform, and here is where the tool configuration lives and the kind of executionist of the process is defined and giving. Now this Paradigm and approach. It's super easy to separate the tooling from the process and to really focus on on the concerns that should be in the in focus.

A

All right and now back to the problem we had at the beginning, as I said, it was really difficult um for quite a lot of end users. We talked to to keep the automation up and running due to the complexity, but with Captain they could reduce the automation code by 90 percent.

A

They had a very nice separation of process and tooling, and additionally, Captain also comes with a set of SRE best practices like SLO validations built in and also important to mention is that Captain does not try to replace your tooling it's more or less an additional layer on top of your tooling, which just takes over the orchestration and takes over responsibility of.

B

A

um Executing those tools, as they should run all right now, let's get back or over to the demo and Mike will look and us what we are kind of doing.

B

All right, thanks, Johannes, all right, so the situation here is we have this hypothetical company named fiber. um They are in the very lucrative Fibonacci as the service industry and it's rapidly growing and so that they basically need to re-architect their system in order to to scale to the passive demand that they're they're experiencing. uh If we look at their current architecture, it's it's quite simple: um the user makes a request. They have basically a monolith application. It does. The processing sends the response back. They.

B

uh They decided that it really makes sense to kind of decouple this a little bit so we'll have the the application and we're going to actually introduce a dedicated Fibonacci calculation service. So that will allow us to scale to the demand that we need uh the way we want to go ahead and do that is we're going to go ahead and build a Fibonacci microservice first, obviously, but we're going to put that behind a feature flag and we're going to at first at least have that disabled, but it will be enabled for the automated tests.

B

So that's going to allow us to test that before actually users see what the impact is. First, we're going to run it through a staging environment, we'll use Captain for that to orchestrate the the staging environment. If that all looks good, we'll move on to the production environment and if everything passes we'll go ahead and enable that feature all right and then quickly just on the right here, you can see what the feature flag itself looks like. uh Basically, what we're doing is we're. You know.

B

If you look at the used remote FIB service, that's the flag identifier. It uniquely identifies the flag in the system. uh The false just represents like the default fallback value. Just in case something goes wrong. That's what the the behavior would be and we're also using the user agent actually is like the contacts that we're providing to the feature flag and in our situation we're using k6 we'll get into that in a little bit. But if the user agent has k6 in it, they see the new feature. Otherwise, it's just disabled our demo time.

A

All right, as now Mike said, we are now going to deploy a new version of our micro service. But before doing so, let's just take a look at our um awesome Fibonacci service that our end users are willing to pay for it's that one. uh When they go there, they can basically run this um this functionality in order to derive the number of 40.

A

and now, as Mike said, we want to update this particular microservice and for doing so, I now jump over to my IDE, where I have everything prepared and I'm now going to trigger a new captain delivery, meaning that I'm now yeah giving Captain the responsibility of for of doing the the delivery of this new microservice.

A

All right everything got started and now Captain is taking over responsibility of doing the orchestration. We can see this by going to Captain and when I do a refresh of this screen, we.

A

A

We can see that the delivery are kicked in and now we are deploying a new version of this microservice in staging when I now go there. We see that the deployment already finished and therefore we used Helm for doing the deployment, and now the tests are executed.

A

As Mike said before, here we are running a k6 tests and those are configured in a way that we are you testing the new feature which is behind the feature flag, but all the other functionality is kind of hidden and to test they should be executed in a couple of seconds.

A

Promise you this will take not too long.

A

Here we go, the tests are finished and the new micro service is also released in the staging environment, meaning that everybody who has access to staging can now access this new microfit version in the meanwhile. We also trigger.

A

Ing the deployment it's.

B

A

A

Instead of waiting for the test, results, I want to tell you or explain each.

A

That's it plus.

A

Let's use that one yeah uh important to mention is that each Captain project has a git repository behind where configure configuration is stored. When we now take a closer look, we find one folder for our Fiber service. This is the service that we are currently updating and in this folder we see that there is a Helm chart, yeah a regular one, and there is also a folder for the k6 tests.

A

Oh all right, while we take a look there, let's jump back uh to Captain, and here we can see that the delivery field mic. What happened now.

A

All right, uh let's now dig into uh the problem itself, because we also got a select notification that delivery now in production failed, and we can also take a look now into Jaeger in order to figure out what actually went wrong.

B

All right perfect so now uh now that we're using open feature with uh and actually uh an open, Telemetry hook, we're collecting metadata on a trace. So we can look at and see what basically happened here uh in this case, we're just going to look at the the Fibonacci requests and we'll see that all of them are failing. If, if real users would have been hitting the site, we would have seen successes here, uh but basically the tests themselves were configured in a way where they would fail.

B

Just looking through here, real quick, we're seeing actually, unfortunately, it looks like the call to the remote service must have had a configuration issue. So even though it passed in the lower environment, it looks like there must have been a misconfiguration in production, so I think we should go ahead and try to fix that.

A

Oops I think I made a mistake because I forgot to update the password before I started the uh the demonstration, but this is a fix. We can do really quickly because, as I said, we have configuration stored in the script. Repository I just go there and in this deployment manifest, there is the place where we are configuring. Our highly secure passwords and here made the mistake of using the wrong one foreign.

A

Of the config, we are now also what we need to redeploy our new microservice in production and therefore I'm using again a captain for doing that and in this case I'm targeting and hitting directly production. Since everything went well in staging before now. Let's execute this again.

A

And we can see that now, as before, the deployment in Broad is ongoing, as I said before, it's Helm, who does the deployment, and now the k6 tests kick in.

B

All right, since the test take a few seconds I'm just going to really quickly. uh You know, peek behind the curtain a little bit and see see how these uh feature flags were configured um so remember it excuse.

A

B

The uh the used remote FIB uh feature flag is the one we were using and then you'll see in here that for everyone it was off, except if you happen to have k6 as your user agent. So this is something that can be done in a lot of feature flagging tools. This is just you know how a flagged chooses to represent this configuration, but it's it's really an interesting and pretty powerful pattern to build a Target uh user properties to make these different decisions in your feature flagging system.

B

So if we check back here, tests are still running so give it one more second. Hopefully that was the last configuration issue that we had uh if we're good to go there, what the next step's going to be.

B

If we wait, um it's basically uh Captain's going to open up a pull request. So hopefully we see this in just a few seconds. If the tests pass, it's going to open up a pull request that that enables that feature for everyone. So that's what you're seeing here uh it says, enable feature it's just pending now. If I were to go back to uh our pull requests, we should see that we have a new one.

B

um Is this the right thing.

A

B

What do we do all right.

A

B

All right, at any rate, we'll look at one of the old ones there we go all right perfect that just took a sec cool. So what we see here actually is is uh Captain, went ahead and opened up a pull request. um It says, basically everything looks good. The tests all pass we're ready to enable this feature for everyone. If we review the diff real, quick, we'll see, all it's doing is is removing that targeting role, we don't need it anymore and we're enabling that feature for everyone.

B

So if this is what we want, we can go ahead and hit approve uh once that's good to go. We can go ahead and merge it and then, if we go back to Captain, we'll see in just a second that basically an event was sent from GitHub saying that we've merged this pull request and it will continue with the the feature release for everyone.

B

So there we go. We see that everything's green and we're good to go right. I think uh we're going back to the submarine now all right. um So let's go through a quick recap here. So basically, uh we went into our staging environment at first uh orchestrated everything with Captain. We did a deployment using Helm. We kicked off some tests using k6 those those tests again were specifically enabling that feature flag only for those tests.

B

uh If everything looked good, we went ahead and enabled that test for everyone and then went ahead and released that feature uh to the wild in production. Keeping in mind. This is the same. You know code there's no, no code changes. It was just a configuration change between those two environments. uh The deployment went fine as expected. When the test kicked off, we did have a failure. That failure happened to be a configuration change, so we were notified through slack.

B

Looked at it looked in open, Telemetry and Jaeger specifically uh saw that it was just a 401. We just had a password issue, went ahead and corrected that issue, and then everything went through just fine. So the the core idea there is. We made a a pretty major architectural change, configuration typos. Those things happen, so this could be a way to you know, add significant functionality or architecture changes in a safe manner, even in production and yeah.

B

uh Let's uh let Johannes do The quick, summary of Captain and then uh yeah, we'll uh we'll wrap it up.

A

All right, the other, mean main key takeaway from uh this presentation. Is that whenever you have problems or challenges when it comes to complex automation, pipelines, think of or take a look at Captain, which is an orchestration layer, helping you to separate from the tooling from the process in order to have a clear to have clear concerns and to focus on what um you want to focus on and when it comes to open feature.

B

Yeah with openfeature, basically we're enabling standardization and uh in Vendor agnostic, feature flagging. So it's just a way to really take advantage of these really interesting and Powerful Concepts in a safe way that works across. You know, vendors and even your homemade solutions, if either of those projects are interesting, which I'm sure they are please go ahead and scan these QR codes or visit our website. Both projects also have Twitter accounts, so appreciate any support uh you guys could provide there.

B

We both you know both projects also have you know very active communities, so definitely feel free to get involved. um You know love to have you and if you have any thoughts on the session, please go ahead and scan that QR code. Let us know and then feel free to chat with us after the session. So definitely appreciate your time and safe travels uh going home thanks. Everyone.