Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2021, 14 May 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Live Experiments with K8s Applications: Pitfalls and Ho... Fabio Oliveira & Srinivasan Parthasarathy

Description

Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon North America 2021 in Los Angeles, CA from October 12-15. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Live Experiments with K8s Applications: Pitfalls and How to Avoid Them - Fabio Oliveira & Srinivasan Parthasarathy, IBM Research

Your K8s apps are instrumented for observability. You are using ingress controllers/service meshes in your production K8s cluster and can shift traffic between different versions of your app. You wish to take your CI/CD to the next level by introducing metrics-driven automated rollouts using live experiments like canary, A/B, and A/B/n comparisons. What could go wrong? We demonstrate how subtle differences in the design of the experiment---how metrics are collected, queried, and used; the traffic shifting policy; the number of requests sent to different versions during the experiment and its duration; and when/how it is terminated---can lead to dramatically different outcomes, and in turn, directly impact the version of the app chosen to run in production. We also discuss simple and statistically effective remedies for the above problem, so that experiments become repeatable and their outcomes are more accurate and trustworthy.

A

Hello: everyone thanks for joining our session today. My name is fabio, olivier and I'll, be presenting together with srinivas and partaserati.

A

Let me start by saying a few words about us. We are passionate about what we call live experiments because we believe that live experiments can help organizations answer the phone question. How do you know? How can you know whether or not every single new code release that you make deliver that delivers value to your customers and to your business?

A

We co-founded a project called iterate that embodies our vision for a live experimentation of kubernetes applications and how we can help businesses and we have experience creating solutions to help sres and to automate devops practices before we go into the specifics of live experimentation.

A

Let's take a step back and look at current trends uh when it comes to cloud native code release practices, these trends, we extracted them from a recent cncf survey report that was made available publicly in november of 2020..

A

That survey found that the majority of organizations release code weekly or more frequently and that there is a widespread use of ci and cd tools in production, but those findings do not come as a surprise.

A

That survey also revealed that there was a seven percent increase in the number of organizations that release code daily, sometimes multiple times per day, which is also not surprising, however, and paradoxically, that same survey uncovered uh a trend that was surprising to us from 2018 to 2020.

A

There was a big decrease, a 21 decrease in the number of organizations that adopt fully automated release cycles. So organizations seem to be raising code more frequently but automating less. That is not intuitive, so why? Why is that happening?

A

There are a few uh explanations for this phenomenon. Perhaps there are newcomers that are just trying out automation, solutions to see what works. Maybe some organizations have the need to control uh a few aspects of code releases or perhaps there's a lack of trust in the tools currently used for cloud native release robots?

A

What can we do to change that? So we are on a mission to change this state of affairs. So here are the goals for this presentation and how it relates to the trends that we that we just saw.

A

First and foremost, we want to raise awareness of facts, misconceptions that we have seen in pitfalls that organizations might encounter when um adopting uh fully automated solutions for uh code releases. We want to elevate the practice of release automation in this process. We will introduce the notion of live experimentation of kubernetes applications and show cases where it can bring value to your business.

A

We will frame common practices such as canary release, a b abn testing, conformance test, dark launches, etc as live experimentation, and we will show you a live demo of live experimentation that is applicable to any kubernetes kubernetes based stack from this point on the presentation will be divided into two parts in part. One. We cover what we call principles of trust body, release, automation and, on part, two we're going to see live demo a live demo that embodies best practices for fully automating release code release cycles.

A

The first thing to notice is that if you want to adopt a fully automated solution to code release, it must be data-driven.

A

uh You want to know whether or not your release is going well or not, and why so, therefore, in order for you to trust a solution, that solution must be able to interpret the data properly so that you can understand what is going on every step of the way, and there are two important properties that must be satisfied by a solution that is providing you with a fully automated release code release.

A

Those properties are accuracy and repeatability. Let's talk about accuracy, first suppose you have an oracle. That knows, for instance, that your canary release is perfect, is good and satisfy your acceptance criteria. An accurate solution would consider that canary a success.

A

Similarly, if you have an article that knows that across three versions in an abn scenario, version two is the best one, then your solution must tell you that that is the version that is the winner now repeatability is equally important. So if you run a robot today do a while today or run an experiment today and get the result. If you run it tomorrow, with the say exactly the same code, you must get the same result.

A

If your solution starts flip-flopping today, it will tell you that the canary is good, and tomorrow it will tell you that canary is bad, even though the code hasn't changed. Then there's something not quite right and we did some research and we observed that some solutions to progressive robots.

A

They suffer from the problem of not meeting both accuracy and repeatability.

A

So what does a solution need to satisfy those two properties? First and foremost, it needs to have statistical rigor.

A

It needs to consider all data points, all observations since the beginning of a row out or an experiment, so that it does not make premature decisions about success or failure, or which version is the best, and it needs to be capable of using the data to perform sound assessments of all the versions considered. So you look at the accumulated data and make sound judgments and assessment you in that those assessments get updated every step of the way.

A

Finally, statistical rigor empower is a solution to be re to be resilient to high metric variance, let's say, for instance, you care about latency as one of your uh metrics for um for uh your when for one of your acceptance criteria.

A

So if the latency is all over the place because of conditions externally to the code, you still want your release, automation, solution to do the right thing and not be sensitive to that high variance.

A

That brings it to the next point.

A

A solution needs to be able to distinguish what is noise from the cloud or what is a result of resource constraints and what is actually caused by the code behavior.

A

Finally, a solution is to be able to adjust the traffic split across all the versions considered, based on the statistically correct version assessments that he makes every step of the way.

A

Now, it's important to remember that organizations don't really release code just for the sake of releasing code, there's always an underlying business need, and when you release code, you actually have an opportunity to deliver business results as you roll out new versions.

A

In order for you to be able to do that, you need to consider two things number one. There is your success criteria in terms of service level objectives.

A

uh Metrics, typically, uh you know people organizations might rely on mean latency tailor, latency error rate, exception rate, etc. In addition to that, you need to consider business rewards metrics such as those related to user engagement, for example, conversion rate and those related to revenue monetary metrics.

A

Now, in order to consider these two elements that uh enable organizations to deliver business results as versions two or more versions are rolled out, a solution to release automation needs to enable the following question: to be answered among the versions that have acceptable performance, which one actually benefits the business the most, and that is the winner version.

A

In answer that question, a solution needs to separate two key elements that affect user experience and they usually are conflated and they should not be conflated. One is how attractive each of the competing features are to your users, and the other is how the versions of the applications are performing. With respect to your slo. Those are separate concerns that need to be taken to account together, but also separate the proper and in order to learn, as you are rolling out and earn.

A

As you are ruling out, a solution needs to be able to perform uh progressive traffic shifting throughout the entire process.

A

One note about terminology: uh we have observed in the cognitive community that sometimes the term a b testing seems to be misused. Sometimes it's used interchangeably with canary release and sometimes it is equated to traffic shifting via http, headers and cookies. Now that that traffic shipping shifting is just a mechanism, it's not a b testing per se.

A

Let me draw an analogy, a culinary analogy here. So if I tell you that I have water flower and yeast, uh can I tell you that I have bread? Do you believe that I have bread? All you know is that I have ingredients. I might not. I might not know what to do with that.

A

So, similarly, just because one can do traffic shifting using http, headers and cookies does not mean that ap testing is actually supported. There is more to be testing than traffic shifting traditionally a b testing is it's thought of solving the following problem.

A

You want to compare two orient versions of your application and you want to identify the ones that uh provides you with uh more user engagement or that delivers more revenue, and typically this is done in the funny way. You um you split the traffic between a control group in a treatment group and you get a data. You look at the data. uh Hopefully the experimentation system will tell you which version wins eventually, and then you decide what to do with that information.

A

Maybe you want robot, and this is a practice that has been done in the front-end domain web and mobile applications for for a while.

A

Now in the cloud-native world, we like to think about a b and abn experiment which includes the following things: the ability to incorporate service level objectives in terms of performance and correct correctness metrics, for instance, in addition to business, rewards not only business rewards but in addition to business, rewards the ability to optionally progressively shift the traffic towards the winning version and optionally rolling, rolling forward or rolling back at the end of the experiment when the system actually makes a decision.

A

So what is needed from a release? Automation solution to achieve that? First and foremost, it needs to use the underlying platform to be able to implement traffic splitting and user segmentation you're going to see those two things in action in our demo uh shortly, and it needs sophisticated algorithms for comparing and assessing all the versions. The versions that are competing in nab or abn experiment continuously adjusting the traffic speed based on these assessments.

A

If that's what you want to do, and importantly deciding when the winner, when a winner can be declared with a statistical confidence, those are the ingredients and though this is the secret sauce that enables you to fully to truly support a b in abn scenario.

A

Now we're going to switch to the part two of our presentation, I will hand it over to srinivas and parta serati, who will take you from there.

B

Thank you fabio hello, everybody. My name is srinivasan and I'm going to show you a demo using this tool called iterate that we are developing to enable cloud native experimentation.

B

A little bit about iterate iterate is an open source cloud native release, automation and experimentation platform. So, if you're developing kubernetes applications or machine learning models, uh when you release new versions of those apps or models, you may be interested in optimizing business, metrics, validating slos, ensuring that your versions satisfy slos and protect end user experience.

B

Iterate makes it easy for you to achieve all these goals. Concretely, iterate provides a new communities, a resource called an experiment. You can declaratively specify a b abn canary or conformance experiments using iterate, and it you can, since it's a kubernetes resource that you can create, you can embed iterate experiments as part of your ci cd pipelines. You can embed it rate, experiments as part of your ddops pipelines and what iterate will deliver for you is.

B

It will compare different versions of your applications and it will pick the best version of your application and automate the rollout of the best version of this application so as part of its experiments, iterate uses metrics and it can use metrics from any rest api.

B

So, if you're collecting metrics for your versions in let's say new, relic or prometheus or sysdig or google analytics any rest api that provides metrics for your versions. If you can query it, then iterate can use it.

B

You can use iterate on top of any kubernetes technology stack um the demo that I'm going to show you uses keynative, that's a serverless platform on top of kubernetes, but you can use it for, let's say kf serving that enables ml a serving you can use it with sto service mesh or any other service mesh or any other ingress technology of your choice and behind the scenes, iterate uses ai and statistically rigorous algorithms to evaluate different versions of your applications and achieve progressive delivery and automated rollouts all right to the demo.

B

So the demo that I'm going to show you is an abn experiment and, as I mentioned, I'm going to use key native serverless platform along with istio to do this demo and as part of the demo, we will also use metrics from new relic and the prometheus metrics backend.

B

The demo will also involve use of business metric optimizing, a business metric along with ensuring slos. There will be three competing versions of your applications that will be in this demo and we will also use traffic segmentation and progressive traffic shift. These are all some of the features that you will see as part of this demo.

B

Incidentally, this demo and several other demos are documented on https iterate.tools, so you can head over there look for the demos yourself and do these demos yourself in under five minutes.

B

Okay. So how is this demo going to unfold? uh There will be three competing versions of the application, as I mentioned, v1, v2 and v3, and we are going to do the experiment only for a particular user segment, in particular for users from wakanda.

B

So if you're from wakanda, you may see v1 or v2 or v3, but if you're not from wakanda, if you're from the rest of the world, then you will only see b1.

B

This experiment does not even affect you, and the goal of the experiment is to find a winner, and there are two things that I want to make sure while finding the winner, I want to ensure that the winner satisfies silos, in particular latency and error rate slos, and I want to ensure that the winner maximizes user engagement, so if there are multiple versions that satisfy slos among those versions, I want to pick the one that maximizes user engagement.

B

This is the winner of the experiment and, let's say b3 is the winner of this experiment and what I would like to do also as part of the experiment, is progressively shift traffic towards v3. So I might have started at five percent for v3, but as soon as I know that v3 is winning. I would like to start increasing it. Let's say move it to 25, move it to 45 and so on eventually make sure that all users end up with v3.

B

um This is how the experiment will be automated by iterate so manually, I'm going to deploy b1, b2 and b3 in this demo. I'm just going to use cube ctl, but in production you'll probably end up embedding these things as part of your ci cd pipeline or github's pipeline, so deploy v1 v2v3. Those are my application versions. I will also use an sda virtual service that will achieve traffic segmentation. For me, wakanda vs, rest of the world traffic segmentation.

B

I will define the iterate metrics that will be used as part of this experiment and I will launch the experiment once I launch this experiment. Iterate will periodically query prometheus. It will query new relic, get the metrics for each version, assess the versions determine a winner and over the course of the experiment, progressively shift wakandan traffic towards the winner. This is the process that is automated. For me, the evaluation, traffic shifting and decision making is what is getting automated for me through the iterate experiment, all right.

B

So with that introduction, let's head over to the actual experiment instructions um I am going to start by creating the different application versions.

B

Let me go ahead and create the istio virtual service that will enable traffic splitting.

B

Let's look at what this virtual service delivers for us.

B

Okay, so, as I said, the rest of the world is unaffected by this experiment, so if you're, not from wakanda 100 of the traffic is going to end up with v1.

B

However, if you are from wakanda even then right now, all the traffic is going to end up with vmware, because the experiment hasn't even started. So as far as all users are concerned, the only version of the application that I exist is v1. So this is the state of the affairs right now in production, these versions will receive traffic from actual end users, but for the sake of this demo, I'm just going to simulate traffic. I'm going to create some synthetic traffic right here.

B

Okay, let's go ahead and define the metrics that will be used as part of this experiment.

B

And let's look at the metrics, we just defined notice that I'm just copying and pasting these commands and these are really cube. Cdl apply in different various forms of cubes uh kubernetes resource creation and manipulation commands all right. So, let's look at the metrics we just created here is an iterate metric and it's called user engagement. This is the business metric that we want to maximize and I am getting this metric from new valid, so I have to literate to fetch new relic metric called user engagement and I've supplied it.

B

A query template that pretty much looks like a new relic rest query.

B

Similarly, I have defined other metrics, for example, the mean latency metric this time, I'm getting the metric from prometheus and I've defined a prometheus metric query uh and template and uh iterate will fetch this and other metrics from prometheus.

B

Okay, so I defined the metrics. Now I'm ready to actually launch the experiment.

B

Let's go ahead, launch the experiment and let's take a look at the experiment specification. This is an iterate experiment resource and, as I said, I am interested in performing an abn experiment. So the testing pattern in this experiment is abn. There are other types of testing patterns you can use, but this one uses abn.

B

There are three versions that we will use as part of this experiment: b1 we do e3, they are specified here. Let's look at the criteria that will be used to judge these different versions.

B

First of all, there is a reward criteria. The user engagement metric that we just defined is what we will use as the reward, and I would prefer reward to be high. There are also the slos mean latency tail latency and error rates are the metrics that I will use for defining the sls. When I define an slo, I can specify the metric, and I can also specify upper limits or lower limits associated with the slos.

B

um Interestingly, there is a section in the experiment called strength, so I can specify here as part of the section I can specify things like how sure iterates needs to be before it declares a winner. So here I'm specifying that iterate needs to receive 100 requests for each version before it can confidently declare a winner, and not only that iterate needs to be 99 sure that it is, uh it has found the best version before it can declare it as the winner.

B

So this is the way for me to ensure statistical, rigor and robustness as part of the experiment, um there's also the duration of the experiment itself. I've said that the experiment lasts for 100 iterations and there is a success between successive iteration there's, a gap of five seconds. So there's approximately a total of 500 seconds is the duration of the experiment.

B

Okay, so let's go ahead. Take a look at what the experiment has done so far. So first of all, experiment is just another resource, so we can first of all watch the experiment resource okay, so experiment is proceeding so far. 13 iterations of the experiment seems to have completed now, 14 and it's running.

B

Let's also look at the traffic split. uh Remember everything was going to be one at the beginning, because you know the experiment: hadn't started. Let's look at the state of the traffic split at the moment.

B

Okay, so, first of all, the rest of the world is still going to be one: that's how we want it. The this should not be affected by the experiment. However, if you look at the wakandan traffic traffic is shifting from v1 to v3.

B

uh It seems like somehow iterators determine that v3 is a good version to move the traffic to from v1. V2 is also getting a little bit of traffic three percent, but it's just minimal traffic residual traffic for exploration purposes, uh most of the traffic, all the traffic wasn't v1, but now it's beginning to shift to e3 from v1 okay.

B

So let's also take a look at dsls and metrics in depth. So here I'm using a tool called iterate cdl. It's a command line utility that comes with iterate. That makes it easy for me to take a look at slls and metrics for the different versions, as the experiment is proceeding so in real time. So here the tool is telling me that our 21 iterations have been completed. 22 now and the version recommended for promotion is v3, and why is that? Because, first of all, v3 is satisfying all the objectives.

B

All versions seem to be satisfying the objectives, but in addition, we three is maximizing the user engagement metric.

B

So my definition of the winner in an abn experiment is among the versions that satisfy objectives pick the one that maximizes my business reward and that's exactly what v3 is doing and that's why v3 was picked as the winner by iterate during this experiment.

B

B

The experiment actually featured several of iterate's powerful features right that includes declaratively being able to specify experiments, a progressive shift in traffic, slos traffic segmentation and so on. There are several other features that we did not demonstrate here. You can try it out yourself. There are other types of experiments, like conformance and canary experiments that you can perform. uh There's also other types of traffic patterns that you can use. You can use a user-specified traffic split instead of progressive traffic split. You can fix the split yourself you can use dark launches.

B

You can indicate uh you can include slis service level indicators as part of the experiment and iterate will enable you to observe slis. Also, in addition to slows and metrics, you can use traffic mirroring and, interestingly, you can integrate iterate with helm or customize uh or just plain yaml. Json application manifest and use it for a version, promotion, automate version promotion at the end of the experiment.

B

So, as I mentioned, all of these demos are available in iterate.tools, so please do check it out and try out these demos yourself. uh You can join the iterate slack workspace and also head over to the iterate github repro, uh to get more answers to your questions or start other discussions.

B

We look forward to seeing you there. Thank you.