Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2022, 1 Jun 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Case Study: Bringing Chaos Engineering to the Cloud Native Develo... Uma Mukkara & Ramiro Berrelleza

Description

Don’t miss out! Join us at our upcoming hybrid event: KubeCon + CloudNativeCon North America 2022 from October 24-28 in Detroit (and online!). Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Case Study: Bringing Chaos Engineering to the Cloud Native Developers - Uma Mukkara, ChaosNative & Ramiro Berrelleza, Okteto

Though Chaos Engineering started as a solution for fixing unknown problems at scale, it has evolved in recent years into a totally different practice area. It is now beginning to play a major role in CI/CD apart from Ops and figures as an aid that improves developer experience. Chaos frameworks are beginning to feature in the list of must-have dev tools. In this session, we discuss the role of Chaos Engineering in stepping up the cloud native dev experience and how developers can use cloud native chaos tests to verify the resilience of their application even before the code is merged. Okteto is an open source tool that enables developers to deploy development environments directly in Kubernetes. The community behind Okteto has succeeded with the idea of providing cloud native chaos tests to the developers in their toolset. In this session we take examples of Litmus chaos tests on Okteto and show how developers can run them as part of the development process, rather than just on CI.

A

Hello, everyone good morning, thank you for taking time to be here great, to see all of you. Today we are going to talk about a new topic of interest chaos, engineering for developers.

A

I am um head of chaos, engineering at harness and I'm also co-creator and maintainer of the litmus project, which is a cncf incubating project, and here we also have ramiro hi everyone.

B

Happy to be here, my name is ramiro verayesa, the founder and maintainer of octeto, which is a cli tool for remote devon vitamins on kubernetes.

A

We are here for the next 30 minutes going to talk about. How can you apply chaos, engineering for the cloud native development ecosystem so before we do that, let's quickly talk about what's chaos, engineering and a brief introduction to litmus project?

A

What can you use litmus project for and then ramiro is going to talk about how some of the developers on their development ecosystem or the cloud platform are using litmus to actually do chaos testing while doing development right? So first of all, we all know that outages are expensive right, we've seen they cause reputational damage financial damage and also, you know, cause a very hard experience in terms of a user experience network motors code. So service reliability is very key for all of us in this era of digital transformation.

A

But we all know that services are always built with redundancy and still failures happen, that's something that's unavoidable and they cause irreparable damage. Sometimes so some of the key metrics for all the businesses are how fast you can recover from a fault. And how can you delay a potential failure right, so you need to be doing continuously whatever possible to get a better reliability out of your code that you are building and you're going to ship.

A

So what's the solution, a definite, a good solution to do this in a very organic way, use chaos, engineering, chaos, engineering is the science of breaking things, willfully proactively and trying to find weaknesses and fix them. You know on an iterative ongoing process and why is chaos engineering more important for cloud native right? We know chaos. Engineering has been there for quite some time, uh we've been using, but chaos engineering is more relevant uh for cloud native because of two things: one.

A

You are really looking at a very small tip of the pyramid when you're building code right there are a lot of microservices that you're now dealing with, uh underneath your stack and they're also changing very dynamically, so just to summarize you're looking at a code in a container which is much smaller, uh that comprises of your service. At the same time, there are so many micro services that your micro service is going to interact with and they're shipping fast.

A

That's the whole idea of cloud native kubernetes enables the system of things getting shipped very, very fast. So in essence you have a code, that's changing all the time in your production systems, so things can go wrong and will go wrong and kubernetes pods. They all get. You know, keeps getting swapped getting deleted for the regular reasons apart. Getting deleted need more need, not be something, you know very bad happened, but they can just get deleted because of some pressures that are happening and then kubernetes reconciles.

A

So this makes chaos engineering very, very important. You have to have a science of how can you build reliability in a structured way while doing uh the cloud native development, and that brings chaos engineering more relevant right all the way from development to testing than production on production, so, in other words, chaos engineering is becoming a kind of a devops culture. It's not necessarily a thing for ops, not necessarily thing for sres. It's still very, very relevant there, but chaos engineering can be on is being adopted as a devops culture.

A

You do chaos, engineering in all places right in ci pipelines in in cd, pre-cd, post, cd and then pre-production post-production. So this this brings a better roi for investing in chaos. Engineering, because you know you are getting your weaknesses from all the way from you know, build to test to production.

A

So how can sres use um chaos? Engineering right? We all know that people start with manual game days and then they try to automate it right. So that's that's one way of looking at it and now you can extend chaos, engineering practices for qa right. So there's a lot of advanced.

A

uh You know procedures happening in continuous deployment, so you can actually go trigger chaos, tests for triggering a build deployment and after the build, is deployed right so pre, cd and post cd, and you can use chaos testing for continuous verification of your deployments that are happening and then shift left you're going even further right. You can actually use in a good, mature chaos system engineering model. Your developers are testing their code using chaos, testing right.

A

We we all know that you know you, you push your code, you commit your code after you do the functional testing, but now you have a chance. Your code can be tested for all potential values. Underneath your stack, you can get it tested right now before you actually merge it. So that's the use case that we're talking about and you can do that using octato as an example. Here, let me quickly talk about uh litmus right. So litmus is a cncf incubating project. We started writing it about five years ago. It's got a great community.

A

uh Thank you all community members for using it and giving great feedback. um It is an end-to-end chaos, engineering platform that lets you collaborate, chaos across all devops. It's not just a tool where you can inject chaos. Chaos engineering is really about collaborating chaos, testing and getting uh the feedback back into the devops right.

A

So it's an end-to-end platform where you have a central api, server or central portal, where developers qa, sres observability teams can all come in and create chaos, chaos, experiments, orchestrate observe and then you know, take feedback and put it back so at the center of it, you have something called chaos. Workflow chaos, workflow is a set of experiments that you put together and you push that into chaos. Hubs right chaos hub is a kind of a gate repository where your team will see chaos workflows as single source of true, and once you have chaos workflows.

A

As uh you know, elements of execution you can target various targets. uh You know most likely start with kubernetes resources, but you can also extend to various other cloud platforms or bare metal, etc, etc. So that's that's a quick introduction to uh litmus, and you should remember that you are basically collaborating around a chaos test or a chaos workflow, and once you build a chaos workflow, you can schedule them whenever you want. That's, usually you know how you do the game days and you can also trigger chaos workflows. You know based on deployment pre-cd position.

A

That's your continuous deployment use case and then you can actually put them into ci pipelines right so and then you can also inject into cd pipelines so most important here for this stock is you can actually cube cuddle them right. So, at the end of the day, a chaos test, a chaos workflow is a manifest. You can that's. That's really is the power of kubernetes. Here you can start creating new use cases for chaos right and once the manifest is executed.

A

It's going to set up the entire chaos environment, execute chaos, and then it connects to the chaos center and then gives you the right results right. So that's how easy it is to inject chaos for developers at the time of development right.

A

So just to summarize, you have an ml file that you need to cube cuddle and that can be collaborated using chaos center and a lot of people can do that. You have a lot of predefined chaos tests. You can, you know, modify them, you can upstream them or you can. You know version them. You can manage this chaos uh workflows and then you can use them in pipelines or you can. You can keep cuddle them in in many different ways. So this is another uh thing that I wanted to mention.

A

It's not necessarily that chaos is, is a left shift process right generally. What we have seen is you, you start chaos in as a game day in sre or you know, ops ecosystem. Then you try to automate them into uh into ci pipelines, but you can always start the other way around. Kubernetes requires more chaos. Testing before the code uh gets shipped, so you can actually start chaos in cinci pi planes, and then you have less uh problem convincing in hey. You know, we've been doing chaos in the pipelines and then we are well tested.

A

Why don't you do this in production right, so that makes it actually more easier for people to do chaos in production, so shift right or right shift for chaos is very much a good use case.

A

So, to summarize, before we go actually and see how you can use litmus to do chaos uh in at the time of development using octato platform. Let me summarize the return on investment for chaos at development stage, so basically you're writing code. You can actually get your code tested against possible faults in hundreds of various different deployments right. You can get it tested on azure cloud. You can just sit down in aws, gcp or other distributions.

A

Before you even actually merge your code. It is possible to get your deployment done in all possible places, cause some chaos and then see whether your code still works, and then you know get there very. Similarly, you can get that done in in your qr testing, as well, so with that let's actually go and see uh how you can do um litmus. Chaos uh at the development environment. uh Welcome ramiro.

B

Hi everyone I'm very excited to be here. uh This is the first live called life talk I do in a while, so it might be rusty, but I hope I hope you do enjoy it.

B

There we go just to summarize litmus. Chaos is an open source skills engineering that helps you validate that your you know your code is resilient. Your applications work octet on the other side, is a open source project. It's a cli that allows you to deploy and develop in remote of environments running on kubernetes when we started octeto, one of our kind of main drivers was that we saw that the divide between dev stage and prod, especially when you adopted kubernetes, was getting bigger and bigger.

B

So it was like three four years ago, when two of my co-maintainers and I decided hey- let's do something about this. We start with this open source cli to launch dev environments into kubernetes, give you code, synchronization, remote execution, all those things as we started to like explore this concept of remote environments, the community started to bring up a lot of interesting use cases, and one of them was hey.

B

Now that we have our development in kubernetes, there's a lot more, we can do as we write code in terms of validation, verification testing than before, because you know, typically, you have to wait until you have a ci workflow to do things like you know, like chaos, engineering, end-to-end testing stress, so a lot of when the community started to poke around and say hey, you know what now that I have a cluster for development and it's remote, so I don't have the limits of like you know, cpu and memory.

B

What else can we do and then I met the people at litmus. We started talking our communities kind of started. Talking, I'm starting to see this really cool use case of doing chaos, engineering as part of the dev flow, instead of something that you did at the end. So I want to show you a demo of how that looks like and and talk about, some of the key points of what we've seen and why why we wanted to share this and why.

B

We think that you know clearing should also be like shifted as left as possible, because it's a great tool for building cloud native applications.

B

So I got a video this time because I don't want the wi-fi to to fail. You know chaos, so this is uh an example of a live environment. This is using octeto, it's running on kubernetes, it's a it's, a simple app. It's a to-do app! You put your your tasks, so you know, as I was preparing this talk, I built it up to track. You know preparing my talk packing my bags getting to kufkon simple app. It has it's a go api.

B

It has a postgres database and you know an ingress. The whole thing and everything is running on a namespace in kubernetes.

B

So now I have supplication, so the first thing that we're doing is we're going to run a baseline to see what's happening right. So one of my favorite things about litmus and about you know harnesses chaos.

B

Engineering is that it's all self-service. As I was saying it's a demo file, you can pre-create that you can put it on your repo, so everybody else on your team can use it. There are catalogs, and one of the things that I really like is that it's completely self-service you don't have to be an expert if somebody else put the yaml on your repo. All you have to do is go to this portal. It can be a shared portal. It can be your own instance load the yaml on litmus click for this demo.

B

I am using hardness's hosted service, but it's the same thing that the open source version, so you create your workflow, you import that give it a name, and then you always have a chance to customize it. So just to show you what we're going to run for this experiment. We have this experiment. That's going to delete a pot we're going to delete the database of our application to see if our application can handle a database outage with litmus.

B

We can pick the target of the experiment for how long to run it and something I really like to do is this concept of probes, which is you're going to be running a test.

B

While the case experiment is running to validate an assumption, in this case, we're going to make a get request to this to-do list to get the to-do items continuously, while the experiment runs, this experiment is going to run for 45 seconds and every 5 seconds litmus will kill the database while in parallel making all these requests- and you know the objective is to see if our application is resilient. Because databases fail, they go down, pots get uh recycled. You know, you've all been there and it's something that's developers we need to be aware of.

B

So you configure your experiment, click start and then litmus will will give you this dashboard you'll, see I've run other experiments before and we'll go. One thing I like about the portal, while this runs, is that it's also very easy to share your experiments and your results with your team.

B

All of this about collaboration about enabling others is part of what makes it easy to shift left these practices, because it's not just about you you're building applications with a team, it might be async, they might be sitting next to you, and- and that is something to keep in mind as we pick. These tools is make make sure the tools enable your team make sure you're not putting extra gates in having one expert run the tests it's much easier and more effective when anybody can participate. So in this case the experiment is running.

B

The first thing that happens is that rhythms will install the experiment on a namespace. It will spin up a part and then the experiment will run if you go to the elitebook scales, give a repo there's a bunch of experiments pre-created. This is the pod delete, the most basic of them, and you can see here now how, if you look at the state of the db, it's processing, because now we have litmus running the experiment and killing the database over and over again, which is not a good simulation for hey.

B

I have I have a bad note in my cluster. I don't have the right tolerations in place and the application keeps getting there and failing- and in this case these are the logs of the actual to-do application, and you can see how litmus is already quitting the application hitting that endpoint and getting some some errors.

B

So once experiment finishes you're going to see the result of it, you're going to see the it's going to end there you're going to give us a table of all the actions that it created. This is a simple experiment. It's only one thing pod delete once you get more comfortable with chaos, engineering, you can build experiments with like multiple multiple things delete delete the pvc create new stuff, but in this case you can see that.

B

Well, you can see it, but right there, it says: probe failed. You can see the better lock next time under that. Oh, I record it with that on there okay anyway. So now we fill the experiment. So what we're doing now is we're going to look at the code you can see. That is the get items code.

B

Super simple, super naive: call: the database don't handle any errors, just get the items and return them. So now what we're going to do is we're going to fix this. We run this experiment. We found that our application is not recipient, and here is what octet oh comes into the picture. This is cli open source. You can get it at on github, you search for octeto and what the first thing we're going to do is we're going to connect our local we're going to. We have our the environment running.

B

So second thing: we're going to do is we're going to connect our local machine with this remote environment, and that is the flow that octo helps you with. You have your local machine, your local ide, all your favorite tools, you run octet up and octal will then synchronize your local machine with this dev environment. In this case, it's going to be synchronizing the code with the api. It's going to give you a remote terminal where you can run anything you want, and the idea with this is that we have a remote environment.

B

We can write code and test it as we go so as you've seen the demo, we could just run a chaos engineering test as we write the code and- and this is what a lot of people are commenting I started to realize- is that without shadow and limbo skills together, you don't have to wait until you write, commit merge, push and redeploy instead octet up then in this case start your process, and then you can run all these tests, so here I'm just showing how this terminal is actually on kubernetes.

B

You see it has the environment variables that kubernetes is injecting for postgres it's running on as in a container. So now I'm we're going to change the code. I pre-wrote this code.

B

I call this venture ribbon chaos and I did something naive just for demo purposes: don't do this in production and it's just retrying for a bunch of times until the database goes back online, and I just did that to show you how we go from an application that is not resilient, because we run this test and we discover this we're going to use subtitle to develop. You see we run octet up. We have this remote terminal into our container. We just run the go process.

B

One of the things that octalo gives you is that you can iterate on your code. You don't have to rebuild your containers every time you don't have to redeploy your application. Every time you can change your code, go run, go run and you see there. The new application is running with my code.

B

What I think is cool about this is that, because everything is running on kubernetes now we can integrate these tools and get this extra level of validation. So we're going to try and say, hey, let's test, to see, if this application works, we don't have to change the send a pr. We don't have to commit and and kind of go through that step. We can just informally write code, make some experiments.

B

Let's say in this case: let's see if this retry mechanism might add, it actually makes sense, and then we go back to libmus and we run the same experiment. We have the jammel. That general is reusable. You don't have to every time, rethink your experiments and this could be a complex experiment. It could be 10, 15 different experiments, one after another. You know like really throwing everything out of your application or something as simple as pot delete, so we're going to rerun the same test again.

B

I did shorten the loop so that we don't have to go through the entire thing again, but I just want you to see how the flow works, launch experiment see it fail, octet up change the code, small changes, you know it on it, see what works see. What doesn't work run your tests again now we're running this against the code. We just modified. It's not the coding staging, it's not the code on a branch.

B

This code is on my local machine and on this remote environment, and then you see how the experiment will start. The database will again continue to cycle, but as the experiment runs we'll see, if our code is now resilient or not.

B

Back to the back to the limits portal back to the results, by the way you can get all of this also through cube ctl, I just like to do it through the ui, because I think that also makes it more accessible to other people, especially people who might not be as experts as we are on kubernetes.

B

But you see there at the bottom now my tests passed like. Actually we have proof here that this change that we made made our application more resilient and then we're ready to go through our formal qiaci pass. We know we have a certain date. It worked. It passes kubernetes it passed on our namespace. Now we can commit push to a branch. Ask for you, know, review from our peers and then eventually make that into into staging.

B

So this is the scenarios that we're seeing now in our community, and this is an example of how you can take all these cloud native tools together and really create this culture of iteration and shifting left. One of our my goals with this talk is for all of you to think what other things you can enable your teams to do before even ci before qa before staging so just to you know, wrap it up our goal here.

B

Is you know when you have ephemeral, dev environments, running on kubernetes, it's easier to run chaos, experiments you deploy this ephemeral environment. You run your tests, you kill the environment, you go to your next task, so this makes it easier it's running on kubernetes, you can reuse the same experiments that you're using in staging in production.

B

While you write code second, is the more: we run these scouts tests, the more we see them, they're, going to start to feel less expensive, because when I, when I talk to the community, a lot of people feel like hey chaos. Engineering is really hard. It takes some time. I have to do all this setup and I get it if every time you want to do it, you have to like deploy a cluster and and get a branch and build all these artifacts. Yes, it looks expensive.

B

It is, but with this kind of tools, because they're more accessible, inclusive, you have cell service dashboards, you can do it more often and then it's going to feel less expensive and more of a normal part of your workflow.

B

The same way that unit tests and integration tests became part of our of our dev workflows and finally- and this is something that I think is important- that we have to think about- which is the more we run this test, the more we use case engineering tools in all the phases of development, the more we use, kubernetes, the better we're going to get at it, because now I I found this bug on my code.

B

I made my to-do list app more resilient now, every time I write code for a database, I'm going to remember all right. I have to think that there is maybe offline and then because I've been doing this, every other application I ever built is going to be slightly better and this knowledge will compose.

B

I will help my friends, my peers and we're all going to get better at it, and that, for me, is, is something that's really important, especially as we move into this cloud native world where, as I was saying, applications are more complex, you have multiple components and, and these things become more important, so just you know to summarize this talk, and you know happy to talk about this after, but let's bring chaos to developers.

B

We want to make chaos a natural part of the debt workflow, let's not put gates to it, let's not think of case engineering as something you only do on qa or something that only your prayer release team. Does it. If you have it and importantly, you know: let's do it, because quality will come up the more we use, skills testing, the more we do. These practices, the more we're developing kubernetes our applications will get better. So thank you so much for attending this talk.

B

um Please join our communities, the links is there a border open source and also there's a litmus channel, and an october channel on the current slack join us. If this is a topic that you want to chat about, we'll be more than happy to collaborate on this and push the state of the art together,.

C

Thank you, folks. Any anyone questions there's a okay, although there's a mic in the middle of the aisle, I'm gonna pass that on the front.

D

Hi, I just wanted to ask if litmus supports some that might create making some network disruption issues or only deleted voters. Yes,.

A

Litmus supports many types of network disruptions network delays. Packet drops network corruptions, uh it's you can just do it very declaratively. Yes,.

C

I'm sorry for those of you who leave the room, please be quiet. We have few more minutes.

E

Most developers that I've come across uh primarily use unit testing on a terminal cli. uh Is there that kind of a capability to have absolutely zero requirement on a web ui to run these types of tests.

A

Yes, so uh I mean that's: that's the real power of what we are trying to bring. So there are certain developers who will go, create a test. Let's say you know a more expert developer.

A

You can use ui to do it, but at the end of it you have an aml file and that gets pushed into your gate right now. Another developer comes in. You just need to do a unit test now, you're going to do a chaos test cube could apply this particular test. You know, along with your unit testing, it gets run from the developer point of view. You just need to do.

A

One command keep cutting apply of that chaos test and internally, it's going to set up all the environment, that's required for chaos and connect to the chaos api server and then give the results back. You can go to the extent of actually uh tagging the results back to your. You know, merge request as well right so.

B

Yeah and to add to that a really cool thing about litmus is that all these experiments are defined in xml, so you can check them in on your repo. You can make them portable, like it supports all these tokens for, like the namespace name, the application target, so you can start to build a repo or even in your application, a catalog of tests for like different levels of chaos, engineering, that's something that I like a lot, because in other in other tools, I've used before the actual effect of the test might be in an application.

B

A web ui, it's kind of harder to understand. What's going on, but the fact that you can put everything you know your helm chart your case engineering. Your development manifests on the same repo makes it that odd a lot easier. It's something that I've been talking. A lot of other maintainers of open source projects trying to kind of get them into like hey, include these tests there I think we're getting there and it's something that I like a lot of litmus of.

A

Course we need more feedback, you know it's a community project. We've been getting a lot of uh good feedback, so please do try it out and then you know create more getup issues. We are happy to engage and then fix, uh as as we learn through the community.

C

Anyone else questions no, don't thank you very much and see you next time.

A

Thank you, everyone. Thank you, have a good time.