Knative Community Meetups, 2 Aug 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Knative July Meetup/ Service level objectives and progressive rollouts for Knative applications

Description

Service level objectives (SLOs) and progressive rollouts for Knative applications

How can you maximize release velocity for your cloud applications, while protecting end-user experience at all times? Service level objectives (SLOs), and safe, progressively rollout of new versions of your applications are two of the best-practices that address this problem. In this talk, we present two demos using Iter8, the release engineering platform for Kubernetes) that shows delightfully simple ways to automate SLO validation and progressive rollouts for Knative applications.

Demo 1 (HelmOps): Dark-launch a Knative application, use Iter8's built in metrics collection feature to validate latency and error-related SLOs, and safely promote (or rollback) the new version.

Demo 2 (HelmOps + GitOps): Progressively shift traffic, and safely promote (or rollback) the new version after SLO validation using metrics from Prometheus.

A

Hello, everybody um I'm very happy to be here. My name is srinivasan, I'm a recent scientist at ibm research, new york and I'm going to be um describing release engineering for k native applications uh using iterate, so we presented iterate early this year kind of when it was still in formation. I would say: uh iterators evolved significantly since the last presentation, so I was thinking.

A

Maybe I could start with the level set of what iterate is uh and also focus on the new capabilities, uh uh the the uh git, ops, friendly capabilities and hound friendly capabilities of iterate. That is, which is what I'm going to present today uh in the talk today, sounds good okay. So, uh first of all what is iterate um iterate is the release engineering platform for kubernetes, and it's really a tool intended to automate slo validation or a b testing abn testing with multiple versions, progressive rollouts with advanced traffic engineering right.

A

So it's meant for devops and uh mlofs teams that are interested in release engineering with these types of features. uh Iterate is really extensible, so you can really use it in any kubernetes environment. Of course, the focus of today's talk is going to be on keynative and you can use iterate with any ci cd gitlabs tool. Also, the focus of today's talk is going to be on helm and also partly on github students.

A

So the basic idea is that with iterate um you can assess different versions of your applications, find the winning version and safely roll out the winning version. So most often you know you have a new release. You just want to check slos if it passes it's the winning version. Otherwise, your existing stable version is the winning version, but you know in other types of experiments, for example, you're doing a b testing or at the end testing you may want to use actually business metrics.

A

uh Whichever version is maximizing your revenue, that's the winning version, so you can really assess your versions in lots of different ways using iterate and do the promotion all right so uh briefly features at a glance.

A

So, as I mentioned, you can use different assessment strategies as part of the iterate experiments, and you can also use different rollout strategies sky's the limit here. So you can dark, launch, perform an experiment and then decide to roll forward. You can do progressive traffic shifting you can do a fixed percentage, split or depending on the kind of networking layer that you have and depending on the level of support that you get from the networking layer, you can pretty much use all features.

A

Traffic engineering features from your kubernetes cluster as part of your rollout strategy and in terms of metrics. So iterate comes with a built-in metrics collection capability, so you can use it to collect latency and error related metrics using you know, test load for your applications.

A

You can also use metrics from any database, uh definitely from prometheus or pretty much anything that is restful that you can query. You can use it as part of iterate.

A

There are also very advanced algorithms behind the scenes to optimize. You know progressive rollouts and traffic shifting, and uh this is also sort of showcased in many of iterate demos already. Okay. So with that background, um let's also quickly jump into the core concept. What iterate is uh and what an iterative experiment is so essentially um iterate crea introduces a new kubernetes custom resource, called an experiment that automates these uh experiments, slo validation, maybe testing, abn, testing and progressive roll-out.

A

Experiments are automated by this experiment, resource and um essentially at the beginning of the experiment. You can perform certain tasks, uh for example, check the readiness of the resources that you're experimenting with during the experiment you can collect. You can, you know, observe metrics, you can assess versions, split traffic advance. You know change the traffic percentages.

A

You can do all this automatically and optionally. At the end of the experiment, you can promote a winning version and there are different ways to do this version. Promotion with iterate um italy also makes it easy to embed uh experiments as part of helm releases. This is exactly the uh point of today's talk, so we call this pattern helmets. It's simply a helm release with an iterate experiment in it.

A

When you do have next experiments, your home values file are required to satisfy a certain schema. It makes it much easier, it's not strictly required, but if your home values file satisfies a simple schema, it does make the experimentation process easier. So we will take a look at the schema also in today's demo.

A

Okay, so let's get started with our first key native experiment right. uh What is what is the problem? We are trying to solve? What are we trying to achieve in this experiment? So you already have a baseline version of the applications running in your cluster and you want to dark launch a new version v2, and you want to do some testing on it. You know just make sure it has a good latency and error rate values and once you're satisfied you want to promote this new version is the new baseline.

A

So it's a simple slo validation make sure your new version is working the way you would expect it to work and then do the promotion. uh Of course, you're probably going to do this in your test cluster also, uh but you know, conditions in your test. Cluster may not always be the same as conditions in your production cluster. So it's independent of what you do in your test cluster.

A

You may still want to do some level of validation before you expose your new version to end users and that's what this uh this release experiment is going to demonstrate.

A

So I already went through the setups by the way everything I'm talking about here is public. It's all documented on iterate.tools, as you can see, I'm driving this entire discussion using the public website. So please check it out. Try it out yourself with your convenience: everything is 100, open, apache, v2 license. You can use it ready so okay, so I already did the setup steps here. So I have helm locally and I have a local kubernetes cluster with mini cube running.

A

I have installed k native in it, and you know this tutorial doesn't depend on any particular networking layer. You can really use that with any kinetic networking layer, so it works with all kineto networking options.

A

I in this particular case, I'm using courier because it's fairly simple to install, but it works with anything.

A

I already went ahead and installed iterate in your in the kubernetes cluster and iterate comes with a command line tool called literacy cdl. That will help me understand the experiments debug the experiments a little bit so we'll take a look at that also- and I believe I added the home repo that hosts many of the sample charts and the experiment charts that this tutorial requires. Okay, so I added the home repo also.

A

So with that setup we are ready to get started. So let's create a baseline version of our application, so I'm going to use help to create a k native uh application. It's version one, it's just a simple hello world application.

A

um Okay, so let's look at what we just deployed: uh it's a simple keynote service. I have a hello world image here and just for fun, I added an annotation that.

B

Says you know don't scale.

A

Down to zeros, if you want, you can scale down to one, but it can be any template service template of your choice. This is a simple example: okay, so let's go ahead and pretend that we have a new candidate version, so we have version v2 and we want to side by side deploy v2 also along with v1. So, as I discussed in this picture, you have a stable version and, alongside you, have a dark launched v2 that we want to deploy and that's what we're going to do right now.

A

So let's go ahead and deploy v2.

A

Okay, we too deployed- and let's take a look again at what we just deployed.

A

So there are a few different things going on here. So, first of all, if you look at the service, uh the revision template within the service has been updated right. So now you have 2.0 instead of 1.0, so you have v2 as the new revision and your baseline version is still getting 100 of the traffic. So all user traffic is really going to the baseline and the candidate version is not really at the moment visible to end users, so it's getting zero percent of the traffic.

A

What that lets me do is do some experiments with the candidate version make sure it's working okay before I really promote it to 100 so, and I'm going to do that using this iterate experiment so, as I said, experiment easy custom resource defined by iterate and uh let's kind of focus on the uh the actions part of the experiment. So this is what the ex the start action is. What iterate is going to do at the beginning.

B

A

The experiment right at the beginning, the experiment. First of all, it's going to check the readiness of your canadian service.

A

It's not going to start the experiment until it is ready, so once it is ready, so I noticed that you know, even though native declares the service as being ready, um the route is actually not available, so there are some dns issues and uh the route wasn't actually available for querying. So maybe it is something to do with my cluster setup or maybe it is something to do with glitchiness and canada.

A

I don't know it doesn't matter, because I can just you know, let things settle by creating this additional task, which just asks iterate to sleep for uh 30 seconds and um then continue with collecting metrics for the new version. um Incidentally, zoom is telling me that audience can now see my screen. Does that mean you were not able to see my screen until now or was it okay.

C

We've been able to see your screen all along. Don't worry! That's yes,.

A

Okay, that's nice, so just zoom catching up, nice, okay! So, okay, my service is ready. I slept for 30 seconds and now I'm actually going to like netflix for the candidate version, and for this I'm going to use the built-in matrix collection capability of iterate and it's able to collect these metrics from it's able to collect latency and error rate related metrics, including latency metrics.

A

So that brings me to the criteria section, so I can actually say that during the experiment verify that the mean latency is within 500 milliseconds, the error rate is within one percent and my paid latency is within one. Second, of course you can change these values, as as it makes sense for your application, but this is an example. These are the criteria that I'm going to use to verify that the application is working as promised as expected.

A

um Okay, so this experiment has already been deployed by health and, let's now see the experiment has probably completed. So let's actually describe the experiment using the iterate command line tool.

A

And yes, so, as you can tell, I just iterate cdl is telling me that the objectives are being satisfied. The slos are being satisfied by the new version. It's mean. Latency is indeed within the limit that you set and so are the other objectives, and if you want to know the actual metric values, you can see that also here. So there is no errors mean latency is seven milliseconds and 12 milliseconds for pay variancy. So everything is looking good, so we are just ready to well.

A

Let's also assert that indeed everything is looking good experiment has completed. Winner has been found great, so we are ready to move it forward. We are ready to promote it as the new version so notice that I'm now setting the baseline tag as 2.0 and the baseline ids feature.

A

So that is the upgrade to the application that I am going to do through helm.

A

Okay, so everything is good, let's just get manifest make sure right. So when I do a get manifest now notice that the experiment is actually not there anymore, because you said candidate equals null, so there is no need for the experiment anymore, because it's now gotten it's now the new uh stable version and as you can see, it is indeed the new stable version. There is no traffic section, which means it's getting 100 of the traffic.

A

So that's a very uh you know: infrared air command line way of you.

B

A

You know this is just an initial demonstration of what an iterate experiment is um so a few things: uh okay, let's actually clean it up check out the application uh you can use this chart actually in production, so you can use the same experiment if you have hand, charts or um general services, you can embed the same experiment or modify the experiment which definitely will want to modify your application templates in helm.

A

So, but you can essentially use this chart by modifying the chart that I just showed you for your usage scenario, and uh there are a few other things. So we are going to look at this more git ops, friendly way of doing the same experiment instead of doing a you know, command line, helm upgrades and there are also a whole bunch of other tutorials for k data we've documented here that do not really require help. These are more raw experiments right.

A

So um the we don't actually discuss the templating part of the solution in these tutorials. You can try them out also, so I'm going to move on to the more detox friendly version of this tutorial, but let's pause here and see if there are any questions, I'm just going to open the chat.

A

Impressive way is very useful to get started understand the tool. Yes, that's exactly why we are documenting it, okay, so any questions.

A

No, let's move forward all right, so this is kind of the same experiment that I just showed here. Once again, we are going to dark, launch a version of your application and we are going to promote it.

A

But here is the twist right, so helen uses, uh as you know, helm uses a values file uh and the values file is now going to reside in get in a github repo and this values file.

A

This github repo will be the source of truth for your application, so in particular, all changes to your application will go through the state people right. So initially there will be a baseline version in the values.yaml file and I am going to manually edit the values.yaml file uh to create the candidate and experiment, and I'm also going to do a manual sync. So we will come back to these manual steps right. So uh the focus is on the interaction with the git repo in this demo.

A

So obviously there are many steps that I'm going to continue to do manually, but they can all be automated with a git ops operator, like argo c. So we'll come back to this. Obviously, integration in just a minute after we go through uh this demo end-to-end.

A

So again, the candidate and experiment uh is going to be sort of uh declared in the values.ml file in the gate, repo, and this is going to trigger an iterator experiment, and the interesting part here is that the iterate experiment, at the end of the experiment, is going to automatically update the values file in the git repo.

A

So the version promotion that you saw me doing in a cli mode that version promotion is going to be done by iterate and it's going to be done by iterate automatically by pushing changes to gate repo to this values.yaml file. That is the flow of this uh demo.

A

So it's essentially the same cluster setup as before. So let's continue so I already have a fork of the repo that I'm going to use and I've cloned it locally.

A

So I'm just going to.

A

The stamp- and let me start by doing this step right, work and clone. Actually, let me start by doing the port inflow, so this is just the git repo that I'm going to use for this experiment for this demonstration.

A

That's my github username and let me clone the repo and let me just create a branch for me to work with this is a branch that we will use for this git ops, friendly experiment, okay, so, let's first of all take a look at the values file in this repo right. Let's take a look at.

A

What's in it, so I have it's kind of the same. Whatever you saw me do with pound set commands, that's kind of what is there in the values file, I'm just going to use the values file instead of the set command. So there is a uh comment section just comment to both your baseline and uh candidate versions.

A

They both use, they have the same application name and the boost have the same repo, uh but they change in terms of their tag and their ids, so my baseline is going to have 1.0 tag and candidate whenever it comes, is going to have a two dot of that right. Now there is no candidate, so it's just the baseline and let's do a sync of this uh heat repel with the kubernetes cluster.

A

Obviously, this git sync is where tools like our cd get into the picture, but for now, for the sake of this demonstration, let me just do the manual sync of the repo with the uh with the cluster okay, so I can do uh once again. I can do a manifest, and this is what we just synced into the cluster. So once again, I have a baseline version, hello, v1, and um it's uh yes, so with image tag 1.0 and that's what we just deployed now.

A

What, as I mentioned, what I would like to do is run an experiment with a candidate and automatically change the values file in the git repair. Obviously, if you need, if you want an external component to touch your git repo, then you will have to give it some permissions to actually modify the git repo. So that's what I'm going to do here, so I already went ahead created a github access, token personal access token, so I am going to create a community secret with that token in it. So let me do that as follows.

A

So now I can create a kubernetes secret um with my github token done, and I can also permit iterate to uh read this token, because it's going to read this token and use it to make changes to the github repo. So let me just give it permissions to read this particular secret that I just created about yeah literally, can now read the github token that I just created all right. So now, let's go ahead and update the values.yaml file so that it has a candidate in addition to the baseline. It also has a candidate version.

A

That's exactly what this is saying, so I have a baseline version, and incidentally, this is the simple schema that I've mentioned in the beginning of the talk. This is the actual helmet schema. There is a common section, there's a baseline section for your basement version of the application.

A

Then there's a candidate section for the candidate version of your application. That's all there is to the schema and obviously, as you can tell, makes it a little bit easy to structure, new releases and experiment with them.

A

Okay, so let's go ahead, update this values, dot, yaml file locally. First and then we will push it to the gate, repo, of course. Once again, this is all expected to be performed by your cd system right, so our ci cd system. So you will have some process to create to build, maybe a tecton pipeline to build this new uh image and publish it to your image repository and once the image becomes available, there is some pipeline that is actually pushing this change to the values.yaml file, indicating that there is a new candidate available.

A

So that is the part that I am really short, circuiting and simulating here by changing it locally and pushing it again. So, okay, so change it locally. Let's push it to.

A

Get and let's also do this call command to verify what we just pushed. So if I.

B

A

A curl now I see this in my values.yaml file in my git repo, so it is, it's definitely pushed the right thing, so I have a baseline version, camera version- and incidentally it's also- uh I can okay. So let's do a let's do a helm upgrade.

A

This is the creation of the candidate version in the cluster. This is the cluster sync step and uh it's also creating the experiment as usual. So whenever I create a candidate version, I'm creating the associated experiment with it.

A

So let's do the helm upgrade and now, let's take a look at the manifest that we just deployed.

A

It's kind of similar the the k native service now has hello v2, but it is dark launched because, as you can tell, the latest revision is getting zero percent traffic and rb1 is getting 100 of the traffic.

A

So let's take a look at the experiment. Once again, it's kind of similar same kind of start actions. It's going to check the readiness of the kinetic surface. It's going to sleep a little bit. You know to ensure there's no glitchiness, it's safe. To start the experiment. It's going to collect some latency at a rate. Metrics uh here are the criteria. It's the same criteria. We used in the previous experiment.

A

What's interesting about this. Is the finish action here right so once the experiment actually completes, it is going to do uh update to your git repel and it will do one of two things right. If the new version is successful, it's meeting the slos it's going to promote the new worship. If the new version is not successful, it's just going to roll back. Then you worship simply going to delete the candidate section. That's what it's going to do.

A

uh I mean we expect this experiment to actually succeed the candidate version to just work and satisfy all the criteria. So, hopefully, what we will see is the actual version promotion, so, okay, so we did this and let's now take a look at where the experiment is at, describe the experiment very good. So, once again everything is looking good. Your candidate is satisfying the objectives and the metrics are all there is. There are no errors and very low latencies mean and talented everything is looking fantastic.

A

uh Once again, it's us right that you know we found a winner. The experiment found a winner and our experiment completed. All good here is the ghidopsi promotion by iterate right. We are not going to do a manual promotion, because iterate has already promoted the version and get so before it did the promotion.

A

This is how your git values file look like and after the promotion. This is how your value spot looks like so before you had a candidate 1.0, sorry baseline 1.0 and candidate 2.0 after there is no candidate, because the candidate became the newbies line and the new baseline is 2.0 v2.

A

So this is what we should really see in git if we check out this file and get right now according to iterate. This is what we should see. So, let's do that check. Oh, what happened.

A

Did I mess up soon?

A

I wonder what happened.

A

This was the step where iterate was expected to promote the candidate version and for reasons unknown to me, it didn't quite actually do that. Maybe I messed up the github token. Maybe we couldn't read it so we can come back and take a look, but the expected uh behavior from iterate is that this is the. This is the change that you will see in the gear ops.

A

This is what I was seeing all along and looks like murphy's law has caught up with me and I'm not seeing it right at the moment.

A

So, let's just give it one more.

A

A

Just verify the experiment.

A

Finished, okay experience team should have finished your case, so maybe I messed up something in the way I set up my secret or something like that. There was no errors. There was no failures or anything during the experiment, so something to.

B

A

My gh, okay, probably so anyway long story short. uh What you would expect to see after this step is this. You would see a promotion like this and that's the version that you will get sync into the classroom um and that's really about it, for this uh ddops friendly way of doing it. So, let's, let's take a step back. There were a lot.

D

Of yeah, can you can you open the github repo with the browser instead of using the coral, because it could be a cache problem? Oh I see that's a very good.

D

um Somebody said that they wanted to see some git ups troubleshooting. So that's how you you're.

A

Right, you're right so absolutely right! I just call again and it's looking good.

A

It is a it is a curl cache problem because I used the exact same logo, man and now it's giving me the right thing, so it did actually do the promotion. This is what you're saying.

B

I'm still kind of interested um like if something goes wrong in between these, like how? How are you supposed to like debug? That's, it's always been something. I've wondered about like get up the get-ups kind of thing it's like. Where do you see that it went wrong and like what would you, what would you have done if that happened, for real.

A

Yeah, that's a very, very good point right, so there are ways to debug the iterate experiment itself right. So the um experiment, as I mentioned, is a uh is a kubernetes resource. So in the status section of the experiment, if something's going wrong, you will see why it went wrong. Experiment will fail basically, and you will see the reason for why the experiment failed.

A

Maybe because you did weren't able to collect the metrics the way or you know get at the metrics the way you're supposed to get at maybe be inside of the metrics properly or maybe because you know the way you configure the experiment with these tasks is wrong, so you will see the appropriate reason when you see the experiment failing, you will see the appropriate reason within the iterate experiment itself.

D

But you are right, uh the.

A

Github token is.

D

Wrong right, it could be like the github token, is wrong. The iterate bring a message.

B

D

In a way, you know one.

B

Of my questions slightly different, um so so, if your main ux is github right, then and and you're letting it all happen in the background, I guess I'm wondering like um you know, if I'm doing everything by pr's, um how do I see that my experiment, like failed, um like you know like or that they're out of sync right, because you've now got out of sync between the two? And I guess I'm just I guess I'm wondering like um what the what the mental model is.

B

If I use this like um like, if I, if I do the experiments, um um how do I kind of see the progress of it and that it failed somewhere? If you see what.

A

I mean right like what is the.

B

Correct: what's the correct workflow, if you're in yeah.

A

So absolutely yes, so there is a notification uh mechanism also that iterate provides right. So at various stages of the experiment, you can actually use a notification to notify anything anywhere. You know any http, endpoint percentage or git endpoint so and.

B

It will contain.

A

Details of the experiment, so it will contain details about where the experiment is, and you know how far has progressed and uh if you will be able to see the satisfaction of the experiment and see that you know this is what has happened so far. It's just yet to happen. So you will see that notification.

A

um The other point, it's kind of interesting that you brought up prs here right. So I purposefully you know for the sake of simplification. I you know I'm demonstrating a git commit here. That's what iterate is doing with the token, but presumably you can create you can use. First of all, you can use a branch where I am pushing to create a pr. That's one thing you can do and the the yeah, so you don't have to directly modify the values file in the end repo. You can do it through a pr for sure.

B

Nice, nice, so I guess if this was like a kind of real production set up and that had happened like would have got like a slack notification or it would have created a github issue or something that's that's. The idea.

A

Right so you you there's already some slack notifications supported by iterate, so you can use just jumping ahead a little bit here. uh There are different tasks that you can embed within the github experiment, and one of them is slack notification.

B

A

Other is the http notification that I talked to about, um and you can, you know, notify your github actions uh workflow and you know trigger it using that or you can use a techno trigger anyway, any whichever you want with this notification or you can use.

D

The cloud event I heard they're good.

A

You can use a cloud event. I think that you, you have a lot of flexibility in how you format the um notifications, so you, I guess you know cloud event- is just a notification format. Certainly you can use a cloud event, format cloud.

B

Event, it's a k-native service running in an iterator experiment. That's what I want.

D

Yes and they go with the progress bar, that's what exactly is asking for the internet, burgers bar right iterate. I see I see.

A

Coming soon, all right, so, okay, so um okay, so so far, let's continue with this, so we're almost there. So it actually did the right thing and, let's sync up the.

A

A

And this is what we get synced right, synced into the cluster, so we have e2 now promoted and running into the cluster. So, yes, a lot of the steps that I showed you right now in this are really manual. However, the key takeaway is that the source of truth about your application in this example is github and that's what is getting changed and all the steps that I demonstrated manually.

A

You can certainly automate it using either. You know you can use auto cd to do you know, pool based need ops, you can use any github's operator, flex, cd or the cd to do full-based, githubs or you can use. You know your traditional pipelines to do more push base below. So you can set up your ci cd system in whichever way you want.

A

This was more about how iterate would plug into your system, and there are lots of different ways you can plug into it. You can use notifications, you can use it comment and you can follow up the commit with a pr. I think that should also be we can extend the task to do it, but I'm pretty sure you can set up your git to create another pr once a branch has a push into it, so those are some of the options.

A

Okay, so everything I showed you today is about you know using helm and using helm in a giraffe's way. um You can also try out some of these other tutorials that are definitely more advanced right. These are there's more progressive traffic. Shifting of you know the traffic going to the winning version and there's also a b testing abn testing, hybrid testing, lots of different assessment strategies and testing strategies, and also user segmentation right. You want to experiment only let's say subset of users using http headers and leave out others.

A

You know that don't disrupt the others, the others just go to the baseline they're, not participating in the experiment. If you have something like istio networking layer, you can do some of these more advanced experiments using issue virtual sales capabilities. So that's also documented already as part of the keynote tutorials.

A

That's really what I wanted to present. Let me pause see if there.

A

Are any questions.

C

While while people come up with questions, uh I'm gonna launch the poll, please help us by answering the poll for this session to to provide us some feedback.

C

In the meantime um we are, I mean we are open to any questions or comments. You may have.

D

I have a question about progressive traffic shifting in k native. We have something called rollout duration, where we we do a traffic, um we take them certain amount of time and with the steps until we move to the new uh this desire uh version, which is the latest. um What is how does if you can talk a little bit about this uh example that you have down there progressive traffic shifting?

D

um Did you take into account like how long does it take to move or iterate? um So the the the the reason that we did, that in canada at the beginning, was like shifting uh to 100 to the new to a new revision, um and you have a lot of demand on the system.

D

You will get a lot of errors so um when we, when, when people do rollouts in these situations, the amount of time an amount of percentage step that you roll out to the new version is very critical and that could change your error rate. Your slo budget in that burst. um Do you take that into account when you say progressive traffic shifting in this case,.

A

So, let's try to understand the questions. Can I didn't know this? The canadian has a feature for moving the traffic percentages. Is that what you're saying.

D

Yeah we we have a something: uh it's called ro roll out duration- um it's maybe I don't know if you call it experiment right now, somebody can chime in uh but yeah you can set it to uh one hour. For example, it's just a one one single config with a amount of time you can say one hour ten minutes and then um it will take that ten minutes and I progressively shift the traffic percentage. Basically just need the traffic percentage to the new candidate to the new latest.

D

um How does your progressive traffic that you want to have in the bottom here? uh Does it sub? Does something similar like that or what? What is the progressive aspect of so.

A

Yes, so it's it's a related concept. That's interesting! Actually I didn't know about this, so there are two differences here right. The way I can tell um when you it's, I think it's the iterate is obviously leveraging the same mechanisms. So when you do a progressive traffic shift, in particular in this tutorial, when you do a progressive traffic shift, it is using the service, the traffic section within the key native service, to do the progressive traffic shifting. So it is underneath the covers.

A

I think it is the same mechanism, I'm not sure how the native native uh traffic shifting is implemented. I.

B

Can jump into this um so so the the rollout duration thing is um it's like very simple, and when we added it, um we we we almost didn't, add it, because we didn't want to conflict with stuff like this, um but we added it just so that you don't have a bad time when you roll out a new revision and you've got loads of load, and you go to any revision.

B

We didn't want, like all the traffic to go to the new one before it scaled up, so it does like basic traffic shifting like when you do a new revision so that it doesn't immediately slam all the load onto the new revision and it does it kind of you know automatically inside the stuff, but it I I I wouldn't see it as like. You know replacing something like this, which gives you like more knobs and.

A

So you see the way one way to think about. It is the experiment that I just showed you there lots of different ways to think about it right. The experiment that I showed you doesn't actually do the progressive traffic shifting it does a dark launch even before any traffic goes to the new revision. It is doing some testing and validation and then declaring it as the winning version.

A

So perhaps one way to think about it is you can do a validation like this then, once you are satisfied, you can do you can combine it with the native uh progressive model of you know the the traffic shifting to gradually ramp up the number of instances of the new revision. That's one way.

D

Once you have your winner.

A

Okay, once you have the winner, the other thing to keep in mind is that the traffic shifting here progressive traffic shifting here is based on metrics and assessment from metrics, right so, let's say you're collecting metrics using live traffic for both the revisions in your prometheus iterate can be used to query that remedies instance and determine who is the actual winner is the stable version you should regain, or you should, should you move to the new version. So it's not just about traffic shifting it's really traffic, shifting based on metrics driven criteria.

A

uh What I showed you was how to collect metrics internally by iterate itself, but the sky's the limit you can use any metric uh back end to query as part of the uh as part of the iterate.

D

D

Using promises right format, standard any promises, endpoint.

A

Yeah, you can use any prometheus endpoint. um You know italy actually has a very, very proper uh custom metric story right, so you can define. There is a. There is another crd that comes with it. There are two crds that iterate defines one is the experiment, crd, the other is the metric cred uh within the metric crd. You can see that you can define a metric, and you can part of the definition is what is the back end from which you're going to get the metric? And how do you? Actually?

A

What are the templates that you're going to use to query those back-ends? So you can there's documented examples for how you create you know these metric objects were prometheus with them without authentication or other back-end services. We have kind of experimented with these other different backends, because there are a few other users and you know use cases where we have actually had the need to explore these other back-ends. But these are just the documented examples, but really anything that can be queried restfully.

A

You can use it as a source of metrics in iterate, new aspiratory experiments.

C

A

All right, so I actually wanted to get a couple of things out of this conversation. Also oneness, uh you know we, the tool is maturing quite a bit so since we first came to the keynote community and presented it obviously, hopefully, if you remember that talk, you can hopefully see how the tool is maturing.

A

uh One thing we would love to do is document something on the keynative side, so we would love to push some sort of a documentation. Pr that you know forget about all the latest and greatest examples, but some simple example of how canadian users can start benefiting from iterate. It doesn't even have to involve helm. It doesn't have to involve anything complex. It can be a simple, your first experiment or release automation, experiment with a kinetic and iterate. So we would definitely want to and love to push something like that.

A

So any guidance on that is appreciated uh and number two is. We would like to also hear uh what are the types of samples that you would want us to develop in the context of key native right, so it is a very, very flexible tool. So if you have ideas on what specific patterns would be beneficial, I would like to hear about that also.

D

um So the first one which I mean the first one um be good to open an issue in the in the docs repo, um and then we have a conversation there, but I I think it's. It should be uh okay, because some people ask about um how to do rollouts and we show how to change the knobs of the traffic splits.

D

But this is could be one example that we can add. It could be a page under admin, but we can discuss it or it can be both. It could be a page, but also it could be a blog post. We also do blog posts, I don't know iterate I've done a blog post, already yeah number of blog posts and also in the game.

A

D

I mean in the canada website.

A

No, no! No! No! Okay! So maybe that's also.

D

The first step, and then the page so because the blog post will be more like um um you know, social type of thing of the. This is a project that interac native, who you are like kind of give a background, and then also so both issues can be open. Go ahead, pretty open to to do that. The other one is I have no. The only thing I can think is the one that jules and me we came up with was notification.

D

So if you can have something around cloud, events of notifying as the uh experiment is going right. So you have a lot of data points for for the experiment: how to capture that into cloud events, and then, uh when one season a broker, then you know the sky's the limit. Do you want to take that data and put it into a database and show a ui uh you want to? You know have listeners to acting on that data, so things that are happening iterate, that are events, facts of life.

D

um If you can make them flat events and send it to the broker, then we can have a story of like extending internet uh based on those cloud events. So one example could be the progress, the progress of the experiment, so it takes an hour and joel's have to have lunch uh launch right all that data points are stored in in in the broker, and then they acted on. um So that's the only one I can think about. I don't know if other people have ideas and then joe's put a message.

B

I I think I said the same thing you said um post an issue in the docs repo, because it sounds interesting to me.

D

Yeah yeah, I think, a blog starting with a blog which is more socializing the thing and then also the page, because the page is like it's a reference and then nobody will like nobody know it exists there. Nobody will look for a searching for it, but the blog post. Is there the social aspect, the value, add and then also the link to the page.

B

I do think it could be interesting to have like I mean you know, this is up to docs, but like have like kind of integrations with a tool section somewhere, you know like how to use as with and then like you know,.

D

Yeah, actually it's it's doing a proposal for k native sandbox. um I know that this project is not in the kennedy sandbox, but the same documentation.

B

D

Could be applied to that right, so the same thing for trigger mesh or other integrations.

B

I think that's a good way of putting it. I don't think we should have a requirement during k nature's sandbox, for us I mean yeah. You can be intimidated.

D

It could be another org.

B

Yeah yeah yeah, that's cool. Let's.

D

Start collecting those.

B

Yeah, the only other thing that came to mind in terms of like demo stuff, I mean obviously like the a b testing stuff um and the current metrics are, like um you know, latency and so on. It'd be kind of interesting.

B

If there was a way um uh you know to do it um with like feedback and and um we and someone was showing us this optimizely stuff the other day and which works for static websites, but you can kind of like see uh how long people stayed on the page right with, like the google analytics cookies and so on um between, like version one with like this header and version two with the other header.

B

um I am sure you could do that with the pieces that you have here and that kind of stuff, but uh it'd be pretty pretty nice to be able to do that with. Like my my web, app that, isn't you know, static, um html and so on.

D

B

I have another.

D

I was so to say: no, we have another metric, um one metric that you didn't you had don't have here. Maybe you didn't mention, but I think would be maybe useful is sometimes we develop.

D

Serverless functions or k native services and a developer would make a change in their code or maybe in the docker file that will make call start break the slo. So I may have an slo that says: no no sk native service should be take more than I don't know a minute to be available, so that metric is available and if a developer did a mistake and now cost starts being it's been three minutes. I don't want to put that in production.

D

So how do I use iterate to capture that and do that experiment and not put it in production? That makes sense yeah.

A

That's an interesting question right. Your calls start will, if it affects the your tail, latency will shoot up if you have a cool start right, that's what is going to happen because the as you query, if you use iterate built-in capabilities to query the um the you know, the dark launched version and there is a cold start issue. Three minutes start up. It's going to show up as a daily latency problem. So when you set the latency thresholds according to whatever is acceptable for your application, and you find that it's not meeting it.

A

This is one way I can think of.

D

Where that code start could be one of those five percent out of I'm saying that I don't care, you know you follow like the ninety five percent. If we do the 95 percentile you're saying that five percent of the whole thing um you don't care about those outliers, but if I want to like care about them, how do I go about that? Maybe change the the percentages or yeah.

A

So you yeah, you could say: oh yeah, I see, I see what you mean. There are a few different ways to cut this right. You can, yes, um you can. You can assume. Let's say you have you're collecting metrics from prometheus right. You can define the percentile as you want right. You don't have to say 95.

A

You can define your percentile as you want, and histogram percentile as you want and say that that is the metric that you want. 98, percentile or 85th percentile 99, whatever makes sense in terms of daylight right in.

B

The right lane, okay, yeah good idea,.

A

Yeah, the other question I think I heard was that I mean that's one approach I can think of for capturing the negative effects of cold start right. You want to actually make sure that we'll start is not too bad, because if it is, then you don't want to promote it. This is the way to do it. You may also argue that you know you want to ignore it. So if there is a cold start, then forget about it, it's not it's going to go away once the you know starts receiving traffic.

A

So maybe it's not an issue for you. I think it can cut both ways. um You can perhaps just send some load test traffic make sure you get past, that cold start hump, and then you can. um You can do the rest of the experiment. That's definitely possible.

A

The other question I heard was about the a b testing and optimizing the use case right. So there is actually an av and slo combined testing experiment. That's documented in iterate that tools by all means try it out. It's not, you know, still not worked out, it's not as clean as the one that I showed you, but so more of a raw experiment doesn't involve helm.

A

uh It is possible today to do a b testing with iterate and q native, and uh perhaps it would be interesting if we come back with a way to demonstrate it using. Let's say, google analytics or something like that. uh This is using a fake user engagement metric. But I guess what you're asking for is: a real user engagement, realistic user engagement metric using something like google analytics.

B

Yeah I mean it could be cool right like um or you know, conversions or whatever, but but yeah. It's that's pretty neat I like that.

D

You have to plug it to actually something that's live.

A

Yes, it should be easy because, as I mentioned, google and any database can that can be restfully accessed, not database, but metrics endpoint. I should say that can be. Restfully accessed can be used as part of these experiments. Definitely, google analytics is one of them, so we can look up some examples that showcase that feature also.

E

Okay, so um I think.

E

um I don't see any other question but um feel free to ask. So um thank you so much srinivasan. um It was great. I I guess this was a great conversation, so we are so happy to that. We had you here today, um please. If anyone wants to present uh feel free to send me a mail, an email and srinivas and you're always welcome to present. You know that so thank you so much for being here and um I hope you we can have another uh meetups anytime soon.

A

Thank you very much. Everybody.

E

Okay, bye thank.

E