Cloud Native Computing Foundation KubeCon + CloudNativeCon North America 2022, 11 Nov 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Turn Me On With Cloud-Native Feature Flags! - Alex Jones, Canonical

Description

Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe in Amsterdam, The Netherlands from April 17-21, 2023. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Turn Me On With Cloud-Native Feature Flags! - Alex Jones, Canonical

Speakers: Alex Jones
Feature flags have long been loved for the ease of turning on or off components running in production. They are fantastic for everything from chaos testing to market research. Open Feature, a new project in the CNCF sandbox, is dedicated to creating a ubiquitous specification for feature flags and enabling them across the entire cloud-native stack. In this talk, we look specifically at the Kubernetes native implementation of Open Feature and illustrate how multiple types of pod workloads can now leverage feature flags. Whether they are web servers, C processes or serverless functions, all can use either Kubeneters native primitives or remote configuration to enable experimentation in a new way. The speaker will walk through key project components such as FlagD and explain the design choices and usage of these services, including an overview of how this enables a complete feature flagging system; with further anecdotes of how multiple communities have collaborated on this project. This first-hand information from one of the project team maintainers will aim to excite and get people thinking about how they can implement feature flags in their own workloads.

A

My name's Alex I'll, be the speaker today outside of my work, I Moonlight, on a bunch of things on the left here: I primarily work for Tag app delivery as one of the technical leads performing due diligence for projects and creating content when I'm working I'm, the director of kubernetes at canonical so I think about things like micro, Cates and upstream kubernetes, and so not an obvious fit to be giving a talk about feature Flags, but I.

A

Think what's interesting is that feature flagging is something that I find very compelling, because I think we have a lot of opportunity right now to take feature flagging beyond the browser where it traditionally lives. So I just want to get some forced participation here in the most introvert way possible.

A

If you wouldn't mind just scanning that you don't have to talk to me, you don't have to say anything. You can just scan that on your phone and we're just going to do a little survey. It'll be 60 seconds. I promise you I'll, leave it up there just for a few seconds when I change slide. You'll see it again. So don't worry if you missed the opportunity to scan the QR it'll be up in a moment.

A

So let's have a look: let's go right back to the beginning. Here, let's start with something simple: how are you enjoying kubecon? My first time in Detroit, it's been amazing, I! Think it's an incredible City, broodless architecture, brilliant people, great conversations good! That's the correct answer: I'm happy to hear that I'm happy to hear that okay! Well, let's move on what does feature flagging make you think of.

A

This is an interesting question, because I think of a lot of different things.

A

Dynamic, that's a good, that's a good answer, a b testing! That's traditionally what I would think of as well a b testing blast radius. That's a really really good idea, deployment Progressive! These are great. These are great Access, Control safety, toggles, etc, etc. It sounds like the audience here is pretty well versed in the idea of feature flagging. So thank you for humoring me on that question and there's one final question: I, just like these two last participants type there we go final question: why do you think feature flagging fits best?

A

There's a few are a few options on here in no particular order. You know: where do we have it? You know I think this is a difficult one to answer.

A

As I said at the beginning of the talk, I felt like until I got involved in openfeature. It was just something you saw in the browser on the server, maybe in your node.js run time or your flask server.

A

Okay, interesting, they're, all spread right, they're all spread, so thank you. I can see. I see, there's also a climbing on client and server and that's sort of what I anticipated most people might say. So, that's good to know so. We've done our slides, so feature flagging is effectively all of the things you described momentarily ago. In that word cloud.

A

It is how you deliver functionality rapidly but safely and exactly as some of you have said, it allows you to do gating or content a b testing.

A

You know think about geofencing, that's a feature, you know, that's a feature, that's being flagged. You might have certain capabilities on your on your movie streaming that are only available in certain regions and they'll. Look at the metadata of your profile and they'll see your email address or your geolocation, or something like that. So petrifugging is really important and you'll see that I put a QR code. I I just discovered these.

A

Before this talk so you're going to see the recurring pattern here, Pete Hodgson put a really great article in uh on Martin Fowler's blog, which was all about feature. Flagging and I decided that, rather than try to do this, this service I would just refer you to the canonical Source, no pun intended.

A

So where does it typically occur so client-side and website and uh and server scientists sort of where you think about it? You know a web browser set aside to client-side rendering doesn't really matter, but it might make a call. It checks. The SDK then checks its future Flags configuration. This is the simplest kind of occurrence. You know you think about it. Sort of like this. You know you can have a an API, an SDK that integrates all that SDI um API, but there's more to it than that. You know.

A

There's caching, there's a valuation there's synchronization, so I don't mean to do a disservice to feature flag providers. It can be quite complex, then you've got rbac users, management, State, etc, etc. But again this is primarily the domain of the browser of the server and then we also have a lot of vendors that build feature flags and these vendors are awesome, but they all have their own interpretation of the world right. A few little examples of what an API might look like or what a function call might look like. There's no particular winner.

A

It's just an implementation. It's how they view the world. It's how they think about evaluating a feature flag.

A

You know you know get my thing tell me if my thing is blue, or is it green as you'll see and so I think that, given that we're in the age of interoperability in the age of bringing things together and trying to build open interfaces, this is perfect, fresh ground for opportunity for innovation, and, if you think about things like service mesh interface, other projects like open Telemetry, who have come together and sorting of vendors they've decided to build something fairly common.

A

It means that users benefit, which is great, so the goals I think, generally speaking, are that Everyone likes our vendor agnostic, agnostic approaches, because it allows the customers to feel reassured that they're not being mopped in, and that enables interoperability, because you can start pick and mixing where things go and, of course, we've talked about this idea of Performing complex evaluations.

A

So what I've ended up building here is effectively a bucket list for what a project might look like, and this project that I'm describing is open. Feature open feature is a brand new cncf project as I'm going to talk about in a moment for feature Flag Management. It allows you to have a singular Consolidated schema and set of providers and sdks that are contributed by the collective of providers.

A

People who are incentivized to build really first-rate experiences for their customers through openfeature and you'll. Hear me touch point back to open Telemetry a few times, because that's one of the more successful in terms of doing this across real world vendors, who really want to make something better for their customers in open source.

A

The API and open feature is simple: it's flexible! It's expensable!

A

It's completely designed by the people who are experts, people who have spent 10 15 years doing feature flagging, who understand their customers as well as people who are first-time contributors like me, who have no idea how feature flags have grown historically, so open feature was a project that grew out of an incentivization to build something, that's better. So that we have that interoperability, it was started by dynatrace, but quickly and rapidly grew to have a lot more folks from both open Telemetry from feature fagging isps and end users.

A

Software delivery companies like myself only this year and if you can believe it we're already in sandbox or a Sandbox project already so from one cubecon to another into sandbox.

A

My participation in all this was based off a chance meeting. I was just larking around in Valencia, as you do. You know, going to parties being at my booth trying to talk about canonical and all the things that we do and I happened to get involved with some of the open feature.

A

Folks and I thought that was really interesting, because I heard about it, a colleague of mine from App delivery, said hey, you should check out open feature we're trying to do some interesting stuff here and, of course, being me, I was thinking well, I, don't do kind of web development I'm not super interested in that I'm, more interested in things, like top of rack switches, hyper channels, kubernetes, orchestration everything, that's not that, and so I started to think to myself.

A

Well, you know I work for a company that builds an operating system that runs on 65 of all kubernetes in the world. hmm What should my incentivization be? What would the philosophy be that I would like to bring to this, and so I thought? What about? We had a sort of a Unix approach to this? What have we sort of thought about running? This is almost like a kernel module as a systemd process. What if we had feature flagging for the shell for C plus for web servers for the kernel right?

A

How would that actually work?

A

How would you build something that could talk to a feature flag file, perform evaluations and then I started thinking about the implications of that so on 25th of May super hyped up on on Coca-Cola and uh and Tapas and and good weather, I sort of started thinking about a single flag D, which is a feature flagging demon that runs more or less anywhere? It's got power, 9 s, 390, arm, 64, arms, 7, arm 64.. It's got about 9, 10 architectures, it supports it follows the Unix philosophy.

A

What's the Unix philosophy yeah, if I was the Unix philosophy which is compactness completeness doing one thing really well, but what it does is it integrates with open feature so open feature? Is this standard? This schema this idea of how you should do things? It's a collection of sdks and it's also a set of providers and it became a provider for open feature. It is a provider, so it matches the API I had some great people that then joined the project. Who are experts on this because I'm not an expert on this I'm?

A

My expertise is on building. You know robust software, but their expertise was actually on the subject matter: So In, This, Very, teeny, tiny demo. You can see that I started running on my Mac and you can't read this because it's like for ants, but what you should be able to take away from this ASCII terminal. Is that I'm showing you flagd running right? So we had this ability in your terminal to do flag evaluation from a file right file on one tab, curl on the other symbols.

A

That wasn't quite enough for me, though, because even though we had this Compact and simple Library, I couldn't help thinking about IPC right couldn't help, but thinking about using AF, Unix, sockets I couldn't help think about other channels and protocols and ways that this could work. And then, of course, when you start extrapolating you're thinking well, why does it have to just work on Linux right? Why does it just all you know who's going to use this another thing right. You know it's a CTL process right now, so I started thinking.

A

Well, kubernetes is something that I do in my day job. What about? We took flag D into kubernetes and I thought. Okay, that's an interesting idea! So I sketched up this design and I'm gonna I'm gonna try and do my best to talk you through this I'm, not sure if it's legible, it may well not be. This is like a coffee kind of scribble that I did on Excalibur, which I promise I'm not paid to mention, but I use all my slides of these guys. So I thought: okay!

A

Well, let's think about how does it actually work? We'd have some sort of agent that's running in a container. That container would need to have, as it would be, a sidecar, because the host process wouldn't want to be bound to it. So I didn't want to I didn't want to compile anything in I didn't want to have it as an SDK that was inside that container, because then there was no way to deliver it right. An application team would have to build flag D into it.

A

I wanted it to be something separate, I wanted it to support RPC TCP, maybe UDP, datagrams I, don't know. I was just thinking about it this time, so it started to coalesce into this new project.

A

So I was on a roll here. I think I might commit I. Think I merge these both into the open feature project on the same day, but the next project was called the open feature operator. So what this did is this operator effectively would run a new cluster. It would look for feature flag configurations, those feature flag configurations. It would then inject via flag D into your workload, and it would do that in a few ways.

A

It would create basically a config file that flag, D would read and flag D, with the help of my esteemed collaborators. Now did some somewhat decent evaluation, so I could save from my flask app go. Get me the current color and flag D would then evaluate that, based on some conditions, it's even more it's even more sophisticated now than it was then now it can do percentile based evaluation. Now it can do all the things I mentioned at the start of this talk.

A

So, what's really exciting there is that we're starting to see some sort of smart logic being able to be driven by flag, D and consumed by your application. In this example, small, as it is, get fruit right, apples or bananas, everyone can relate to that apples or bananas, but the flags themselves could be changed through the crd, so the custom resource of kubernetes would contain the flags I wouldn't have to touch that little config file there.

A

So that was the initial thought. That was the initial thought. We then sort of Road tested this affair and I realized that that was not the way to go. Config map mounting inside of kubernetes is not the way to go, because the the lease Timeout on the conflict map is too long. So, if you want to update it, you've got to wait for reconciliation and it'll, be up to 60 seconds, so I thought we need to put this into sixth gear.

A

We need to put the uh put the speed on this, so what I then went back to drawing board drawing board with was let's use a shared Informer Factory, which is a capability of the golang library for client. Go it talks to the API server directly and says: hey I'm, a pod I'm, this flag, depod I, have a bit of code in me that connects to kubernetes API, and it tells you what I want to watch this API so from left to right.

A

The way this works is that I have a deployment I simply label this end to end on the deployment and I have that crd that matches the name end to end the admission controller goes to the operator. The operator says: hey I can see this deployment wants to create pods that are open feature pods. It then validates that CR exists because there's no good creating there's no Flags to read right. It then mutates the Pod spec, so what it does by mutating.

A

It is this pod spec probably only contains one container, it injects a second container and it sets up the permissions that creates a cluster role binding because of course, our back is a big challenge here, because you're talking to the API server directly, so it creates a cluster role binding. It does all the Machinery to get that working and then your workload container is able to consume it now. The beauty of this is because we also build the SDK, the S, the client SDK, that you get to work with as a developer.

A

You treat it like localhost, always so, if I'm on my Linux machine building, this workflow container I can just spin it up in a CLI and if I go to capes, it's the same experience. I don't need to put any connection strings in I. Don't do any of that. The gubbins all I need to do is say, dot, grpc or dot, HTTP or HPS. Who support the lot, in fact we're using the Golan connect Library, which lets you do multiplexing on the same port, which is quite cool, so kubernetes beautifying?

A

That's only step one! That's for me! That's vanilla! Let's go to something a bit more interesting! What if we could take this further to start building operators that consume Flags operators that actually were able to be controlled with flag d, so I started thinking about well. Here's an example program that I wanted to build. It's called Watchman, it's a it's a validating emission controller and what happens is that I want to be able to turn it on and off. I should be able to pull the plug on my cluster from creation of PODS.

A

It's actually a pretty attractive proposition, if you're in a lockdown mode on a cluster. How else would you do it? You'd have to revoke user, are back or you'd have to cut the API server off or you'd have to find any number of of hacky ways of doing it. But with this you could use flagd, you could trigger it via flag and it would simply lower the drawbridge and the validating admission controller in the in the admission chain would reject it and so I thought.

A

This is really interesting, because I think we're onto something here so I'm going to show you a few demos, because I think I spoke a lot there. So the first thing I want to show you is the demo that some colleagues made using flag, B and using open feature operator. You have the controller manager, that's the operator, that's the governs that I helped put together and then what that does. Is that injects flag D into this thing? That thing is the demo deployment.

A

It is one container, but what it's done there is it's injected flag d as a sidecar I'm sure that you're a very Savvy audience who are very familiar with sidecars, but I thought I'd mention it in case. You haven't used them before. So, let's port forward this demo and see what's going on so the first thing I would like to do is to put the demo this bit on the left. I'm going to put my screen on the right, so you can see exactly what's going on.

A

Let's refresh this cool, this is a very cool little Fibonacci demo, I made absolutely all of it. No that's a complete lie, I mean none of it and it just it's great. So this is really useful to the demo one. So thank you again for helping me this. So what happens here is that if we look at the demo deployment you'll see that you've got the demo, you've got certain flags that are being checked.

A

What happens now is that if I go to the feature flag, configuration I can see that my open feature demo has an end to end certificate Flags. This is really cool. So now what I can do?

A

This is the spec by the way for I can feature the schema, so you can see that we can decide on the variants I'm just doing a very basic example here, but if we go to say line 38 and we change this to Green, save that yellow file, we go out of that yaml file and we save it and what happens in the background? Is it upgrades and it turns and changes to that color now, what's also interesting, if I, let's do a live, demo is always exciting.

A

Oh I think my port forward might have timed out. Let's give it another little whirl. Let's have a check, we'll probably exist. Okay, there we go, demos aren't supposed to work the first time. That's just the rule. Let's have a look.

A

Okay, let's go demo deployment there we go.

A

This is the the nerve-wracking bit when it never quite works. Let's have a look. It's got a flag not found again, like I, said: I didn't make any of this. So I'm just going to remind you now that I did give myself that caveat, uh get up, let's try and have one more little look, and if it's still not playing ball, then I'll move on to the second part of my demo. Oh just as I started terminating the Pod, it decided to to kick into gear.

A

Let's try it one more time so, once let's just get rid of this pod and we'll try it again, we've got a little bit of time. So it's just pulling my image down. Interestingly enough, this cluster is in my shed in South London, it's I'm using tail scale and it's gone through there, and so that's kind of why I have a little bit of latency on here. However, I thought it was cheaper than having to host it on a cloud provider.

A

So that's something for me to learn in the future that the Wi-Fi isn't necessarily the best. So, whilst that's running in the background, we're going to switch the other pile of demo, I mentioned Watchman this controller manager- this is a pretty simple example of an admission controller, quite simply put if I go to Cloud code, Watchman and I go to control, create, apply, config samples, pod, right, I, think it's a simple nginx pod, something like that: I, don't I, don't know what it does. Let's have a look. You can see it's starting up.

A

You can also see I've, probably got some Network latency, because my pod images aren't being pulled very quickly. Yeah there we go. Let's, let's kill that. That's a standalone static pod, so no harm done there. What I want to show you is how quickly this updates with like the underneath open feature. So this has a very simple set of feature flags and on off variant. That's what's happening here, so I'm going to change the default variant to on there. We go I'm, then going to go down to here and what you can see.

A

Yes, a demo that works. Finally, you can see that flag is set to blocking admission right. So imagine you're working at a bank and you've got a cluster that is Mission critical. You know that you've either had um you know a bad actor to join that cluster or you've got a developer who's inadvertently, creating resources, that's destroying something or mutating something critical by simply flipping that feature flag, you've blocked access for everyone in the cluster which puts it in a safe space because you can treat it like a box. You can forensically go through it.

A

So this example of Watchman is designed to show you how you can do that with flagging and just to show that it's not smoke and mirrors. We can just go and fix that so here's canines and we'll turn it back on momentarily and we'll see if the other things finish pulling now. This is this is in for the long haul.

A

It is a node.js image, so it's probably about 15, gigs, okay, all right and we'll turn the Watchman system back on and I'm, silently relieved that I chose to do two demos rather than one that turns the variant back off, I'm, not sure if I print anything on the log, maybe I do yeah I. Think I do yeah. So you can see here.

A

What actually happens is I check in the log I'm going to do a bonus round here, since we didn't get to see the other bit of the demo and just show you very quickly actually what the code looks like from an implementer's point of view, so I hope you can see this I have a web hook here in my web hook.

A

I have a component here, so it says, get flaggedy value right that flighty value calls the SDK so that calls the open feature. Client, SDK, there's no connection string. As mentioned you, you can set the type of connection, so I think I do that in the main.go here you see I tell a lie: I did put a connection stream, but that's, but it defaults to overhost. Anyway, and so I set the provider I set grpc, because the office is cool and, as you can see, it's gone through. It's check to see does this exist.

A

This is just in a Timeout Loop and it's then gone and done what I needed it to do. So we go back here. We can then create our pod once more and I promise that that will be the end of my messing around with this all right turn that to off great, let's create our engine, export again uh apply and there we go pod created, exciting cool.

A

So, let's go and just summarize that we didn't get to do the playground application. However, you can do that yourself on your own Hardware that pulls stuff down properly and play around with it. It's generally available. The QR codes I had previously should give you some context as to where to get that from it's on our GitHub under open feature and the second demo we did was hitting the brakes and you need to stop a creation of a pod in a cluster and, of course, that is just this tip of the iceberg.

A

Right just doing something like the emission control is super simple. We can take it a lot further. We can put it inside a module for an Ingress. We can put it inside some other piece of microservice behavior that needs to switch from Services inside the cluster that perhaps he potentially needs to mutate, something else. So there's a lot of applications with that and, of course that made me think what's next right, we're able to now mutate and uh modify the incoming pod spec and put flagged inside of it.

A

That's really great we're able to enable users, but what about non-user-facing workloads? What about system, workloads and so I started another project because I get quite bored easily and I thought I try another one! So this now it's not quite raised for prime time. This is the API server and kubernetes being rebuilt with flag D inside of it. So you can turn parts of the API server on and off dynamically, so feature Gates.

A

You can start to flick on now if you've been a kubernetes admin before you'll know that you have to restart the cubelet and pass new parameters in to enable certain capabilities. It's not actually a code limitation. It's a decision to do that and what I've started to go down the route of is building a project that will allow you to do that in the API, server and I think that's exciting, because we can start to do upgrades. We can start to play around with capabilities.

A

We can start to give developers access to Features without fully committing. We can also go another direction as well. I've mentioned workloads from a Persona of the management cluster I mentioned it from the application workload, but there's also another dimension, and that's actually taking it further down the tree down the stack into the infrastructure layer, top of rack switches, dpus smart mix, starting to look at what needs Network acceleration. Let's say: I have something going on in the data path.

A

I can analyze that and decide whether or not to put it through a network, accelerated node, so putting flag D inside a scheduler could be something that could be quite interesting and the evaluation system which I didn't cover in a great detail, is getting fairly robust. Now it also overlaps my previous talk about data operations with this sort of stuff. You can start the build declarative configuration for different types of failure, mode right so effectively when, when a type of request comes in, you can build failure modes with the configuration flags and I.

A

Don't think that this was ever really something that anybody anticipated with feature flagging. One final thought just on this is flag: D also has the capability to contact the remote API of a provider. So if you have a feature flag provider in the cloud flag, D can connect through its https service and talk to their API hands in the air. We haven't matured. This very much. We haven't specifically done a vendor API, but it's there that all the code's there.

A

So what you could then do is you could turn a feature flag on and off in the browser and it restarts your load balancer or it modifies something in your customer I think that's a very interesting Prospect for democratizing who and what gets run experimentally on your cluster in terms of our roadmap, we're here right, October, November, soon, Christmas by next march, going on April we're going to be looking at maturing the project even further. Our roadmap is going to be shortly released.

A

One of the things I didn't mention is I've just joined as a fairly neutral party on our governing board to try and help lay out this roadmap, because we want people to have the confidence that they can start building production grade infrastructure against this.

A

I wanted to finally share some links here for those of you who are interested in getting involved the website openfeature.dev the repositories, the community. Please do get involved. It's only made as good as the number of contributors and people that want to help us. We've had a really great start. We've already got one production use case coming in fairly soon, and we've got a lot of other folks who are looking at bringing into their organizations.

A

I thought I had a last slide. However, that is me and I'd like to thank you for your attention. Cheers.

A

I think we have time for some questions. If you would like to ask any questions, I'm happy to try and answer them.

A

Can I tell you who's joined an open feature, any vendors, yes, I certainly can we've had some diligent um work done on our community repository already that outlines fairly fairly well interested parties interested contributors, folks who are already working on it.

A

There are going to be no doubt some very familiar names on some of these projects on some of these participants and we look to grow that, in fact, we I think we had folks just approach us at kubecon who are a feature flagging, an experimental feature company who want to build a provider for flag d. If you want to build a provider or an SDK, we have all of this stuff inside of the open, Future Community project, and please do check it out and ask us more questions. We have to help.

B

Hi, sorry, um thank you for the talk in the demo. um You mentioned earlier that there's some complex schemas that were supported by other features uh and I was wondering if you could show an example of that. Maybe.

A

So I wanted to also prefix just by saying it's not my expertise area, so people in the crowd from the project I don't want to do a disservice to it. However, there is a schema that has been agreed upon by the community. It's being revised continuously. I will just outline a very high level example of that of that full schema just for you to see right. So this. This is something that um I know is starting to crystallize. But you know, as I mentioned things like a valuation content doing fractal evaluation.

A

Those kind of things are coming in are in there already and I, wouldn't be surprised. If we look at doing further work in this direction, however, we do take it with a certain sense of um you know, I think responsibility. We looked to open Telemetry as well. We have to be very careful about making iterations on the schema once we've had, people start to build towards it with sdks and providers. But yes again, it's under the schemas repository.

A

We have issues, we all have issues, but we have issues if you want to if you want to participate. Also, if you're interested just in the architecture we have um othebes ofeps are open feature enhancement proposals. So if you want to get involved in the project, you think you know what I think you folks are doing a great job, but I think we should go this direction, get involved. It's it's a great way to keep things going.

A

um I had to answer the question. Thank you anymore.

A

Okay. Well, thank you again for your participation, any everyone that's following up tonight and tomorrow, safe flights and thank you so much.