Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2022, 2 Jun 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Cloud Native Chaos Engineering with Lit... Karthik S, Umasankar Mukkara & Udit Gaurav, Saiyam Pathak

Description

Don’t miss out! Join us at our upcoming hybrid event: KubeCon + CloudNativeCon North America 2022 from October 24-28 in Detroit (and online!). Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Cloud Native Chaos Engineering with LitmusChaos - Karthik S, Umasankar Mukkara & Udit Gaurav, ChaosNative; Saiyam Pathak, Civo

The discipline of chaos engineering has evolved since it was introduced by Netflix a decade ago, mostly as a result of the cloud-native paradigm and the proliferation of Kubernetes as the universal control plane for today's distributed architecture. While the essence and basic principles of chaos remains the same, the way it is operationalized has undergone a paradigm shift, not limited to - the faults themselves, the environments where they are executed, the persona carrying out the experiments, as well as the methods to run them. LitmusChaos is a framework that has been designed to address these newer requirements and enable users to proactively identify weaknesses and improve resilience in their cloud-native setup. This session provides a deep-dive of the project, its goals and how it achieves them.

A

Hi everyone good evening welcome to kubecon again uh I saw some of you in the morning session, so some of it is going to be.

A

Good, it's going to be of a repetitive in nature, um just to introduce ourselves a few of the maintainers of the project. Litmus chaos are here: I'm umam mukhera. We have karthik and udit.

A

All of us are been with the product project from the beginning for about last five years, so happy to answer any of the questions that you may have, and we also have an end user of the project, uh siam patak director at sevo and uh he's also a cncf ambassador.

A

So we have about uh 30 minutes and then we will take.

A

uh You know: q, a in the last five minutes, but for those of you who are new to litmus probably I'll rush through the first five minutes on uh what's litmus is and violet miss and then karthik is going to uh talk through about uh how the journey has been for us um and then what's what you can expect uh the rest of the year uh from the project and also then um you know, how can you contribute um we're, obviously interested in working with the community and taking more contributions?

A

So you know: how can you contribute if you're interested in contributing right and then uh siam will do a quick demo of it? So outages are expensive business reliability, service reliability is very critical to business. You don't want to get into the irreparable damages uh that are caused by outages right and even though your systems are always thought to be built for reliability. You're building redundancy in your systems, um failures are inevitable.

A

Right and failures will happen. If not frequent. It's only a matter of how often they often happen, rather than you know whether they happen or not right. So how do you really avoid service down times? That's the bigger question that we need to ask as we move towards. You know: kubernetes based microservices deployments right. So as part of this, we all we have seen importance of the following metrics uh continuously, being uh you know, talk more and more uh in the business discussions.

A

You know how fast you can recover from your outages, and how uh can you avoid these outages or how can you increase the distance between these failures or outages right, so the answer to that is: adopt chaos, engineering from beginning and as much as possible in your all phases of devops or cloud9, everyday ops and also chaos. Engineering is not a magic bullet. It's a step-by-step method to build a kind of a culture into your devops practices right. You always start with a few small chaos tests and then incrementally start building.

A

uh You know over a period of time, automate them, etc, etc, and why is this more important in the cloud native world? That's because you have more dynamism. You have more containers to deal with and your view of your code is much much smaller. There are so many things that are changing outside your code, so you want to be able to test how my code works when something else fails right. So that's very, very important and this code always keeps changing faster and faster right. So all of us are trying to see.

A

How can I increase my ship time right? So that that's one thing that continuously is happening so together with more number of micro services and the faster shipment, you have a situation where there is so much dynamism and you don't know how to take care of or how to ensure reliability. So chaos engineering definitely will help there. So chaos engineering. Think of this. As a devops culture, it's no one's responsibility.

A

Anybody can start, you can do move left or right, but important thing is that someone has to start doing chaos, experiments in the entire devops and then start training, others right. So that's that's. What is important and latemus helps you do that litmus is a project which is going to provide you a full platform to manage and orchestrate your chaos experiments across your team members in the entire devops right. So it's got a centralized or control plane where developers, sres and qa members. Even your management can come and see.

A

What's going on with respect to reliability in your devops, and you can take actually some control on building resilience into your system right and it is a kubernetes app. It works for non-kubernetes targets. Also right and you can actually execute chaos experiment across various cloud platforms as well.

A

How do you get started with litmus? It's pretty simple: it's a helm chart um you can actually deploy it on any kubernetes. It takes minimal resources being a kubernetes app. You can extend it as you grow with chaos, or we also have uh you know a hosted control plane. um You know litmuscals.cloud just to get tested and then keep going right. So you basically start with a simple installation either by helm or signing up, and then you start connecting your actual targets through litmus agents to your control plane and then start creating your chaos test.

A

Litmus gives you a set of experiments already about 50, plus experiments, and it gives you a good sdk api. You can be very creative, you know once you get a hang of it and then you can start running chaos, experiments and start running analytics, seeing how reliability is improving across across your ca pipelines or cd pipelines.

A

So we got a bunch of experiments. I think you can go to hub.litmusg.io and then start seeing it, and this is one area where we've been. uh You know seeking um uh more contributions right. So, as you learn um chaos and as you uh you know, experiment with chaos, you get better ideas. You can upstream them for the community to get used, so chaos experiments are stitched into chaos, workflows and chaos. Overcloses is nothing but your your chaos test right.

A

So you can actually, once you build chaos test, you can schedule them or you can put them into your githubs right. So, as you deploy your deployments, chaos can be triggered automatically or you can inject them into crcd pipeline or you can just ask your developers hey. You know I have some chaos test. You can run yourself right, you know just do it the cloud native way or cube cuddle way. So it's it's a very stable and uh you know reliable project.

A

We've been uh seeing people using it for last few years in an enterprising way. So it's got a full list of features that you can go and you know learn more about. But uh the next thing is, you know, like karthik and other maintainers are going to actually talk about what has evolved in the last few years and what you can expect uh in the year and you know going forward so with that. I will invite uh you know karthik to take us through the next few minutes.

B

A maintenance session you might all be interested in learning about how the litmus journey started and what were the decisions we made along the way in building a cloud native chaos, engineering platform. So I'll just take you through that, so we built litmus basically in response to a need to test the resilience of a sas platform that we were internally building and we wanted that was communities based.

B

So we wanted to go ahead with a chaos engineering platform that helped us to do chaos on a kubernetes based sas platform and do it in a community's native way. There's a lot of communities being thrown in here. What do we mean by that?

B

So people are used to running operations on kubernetes clusters in a certain way, everything is declarative. Everything is a resource and the resource is reconciled when most of your application, life cycle management, security policies, and so much else is being done in that way, you would also expect would also need the resilience tests to be done in the same way sort of homogenized the way you do. Chaos, experimentation, residency tests and the platforms that were available. At that point of time, the chaos tools were not meeting that purpose.

B

That's when we went ahead and wrote litmus, the idea was to keep it open, source and community collaborated, because that's how people really use their that brings in a lot of rich libraries. You could use that to create faults of different types, because everyone has their own idea of what to fall should be, and then we went ahead and made a plug-in based chaos platform. That is, you could bring your own experiments. That's what you see on the right side.

B

You could just get your own fault business logic containerize. It run it as a kubernetes job instrument, it a little bit and then it will be orchestrated as part of the platform that makes it easy and standardized for all people to come and write their own faults.

B

So the initial deliverable there was creating chaos, experiments as custom resources. You had an operator to carry out the business logic defined within those custom resources. We put all those experiments as part of a hub and byoc is just bring your own chaos just keep expanding the hub.

B

That's what we had initially and then, over a period of time, we got some feedback on how we should improve um people needed a way to centrally manage their uh chaos on their cluster fleet, not gone individually, track chaos, experiments in each cluster, have it all visible from one platform and they wanted to do complex scenarios.

B

Okay, one fault is great, but most of the time you have complex scenarios in your environment. Things are getting built up over a period of time in production, so you're really interested in doing chaos. Workflows not just single faults. Workflows are a sequence of faults, strung together in different order that you need, and then you have the ability to collaborate with your team members on the workflows that you created, store it in a git repository, have a single source of truth and add observability features.

B

Here we are talking about the ability to set certain constraints around what should happen when your default is injected be able to validate. This is how your application behaved during the fault injection. So add we added those features, and then we also added some features around automation. When do you want your chaos to be injected at what point, maybe after a application, got upgraded as part of some deployment by a pipeline or by some git ops controller like argo, cd or flux?

B

So that's the observability, the automation, the teaming and the user collaboration features and then the workflows. These were the major additions that we made and all of them from one centralized control plane into which you can correct different clusters that are the target environments for chaos.

B

So that's the evolution and, if we've sort of ended up with this there's a chaos control plane, which is where the users come in and say what they want to do as part of their experiments, create their workflows, manage it collaborate with their team members and, on the right hand, side. You have the execution environment where the chaos actually happens and we've got metrics that you can use to instrument your own observability dashboards. To say this is how the application is behaving when a particular experiment is running.

B

So at this point, I'll, probably um just summarize what I just said- the original principles. You saw that in the first slide, the ones they are the ones on the right hand, side of this screen and, on the left hand, side. We added certain observability capabilities with probes and exporters, and things like that and added the ability to run workflows, schedule them, integrate them with your gitops flow, etc. So this is sort of broadly the framework for cloud native chaos.

B

Engineering at this point that litmus sort of defined- and this has found some resonance within the community.

B

I'll, probably let uma talk through some of the use cases, the way it's being used most commonly, and I think we'll go back and speak about the roadmap after that.

A

I'll, probably in the interest of time I'll just rush through, uh we talked about this slide in the morning as well. um You know the the main point that we need to take from here is. Chaos is also for qa and developers right. So the way litmus is architected.

A

Is you get a declarative file, a yaml, that's well uh tested by somebody in your devops, and then you can actually go and use it elsewhere, uh wherever you see fit right, so think of chaos, testing spreading across devops and there is incremental feedback being built either shift left or shift right into that right. So that's, that's!

A

Probably uh one of the things that I wanted to share and you can throw uh into game days or ci pipelines or cd pipelines, and one of the most common things that we have seen happening in litmus is that being used along with gitoffs right people uh trigger chaos.

A

As a task of um you know, deployment happening um as part of your cd cd pipeline, so you you can easily automate it either using orgo, cd or spinnaker or flux. It works with uh it's been tested by community and uh you know proven to work with most of the cd platforms as well. So uh some of the other things that we have seen litmus being used is also to test observability. You have invested in your observability systems and you don't know whether something goes wrong.

A

Your observability system works well or not right, so you can actually introduce chaos uh and uh see whether you are getting the right alerts or your people responding to your uh you know, uh setup practices uh to such chaos or not right. um We're also seeing people testing it uh kubernetes itself. Kubernetes keeps changing all the time right. So when you upgrade your kubernetes stacks can actually verify your entire application against uh possible failures happening in kubernetes itself.

A

For example, when a cubelet service goes down, you know, or you set up for a proper service redundancy or not. We are also saying um kubernetes being used as a cross cloud control, plane and you're, bringing in a lot of infrastructure into new infrastructure into the consideration. So when such infrastructure fails, how does your service work right? So how can you inject failures into uh some of um the cross cross cloud control plane and then we are also seeing um you know. Kubernetes deployments are always hybrid in nature.

A

There is some stack that is not kubernetes underneath so what happens to your service when failures happen under such stack? We're also seeing people taking chaos all the way to your well-defined services, such as you know, amazon, rds or other data services. Can you actually simulate chaos against such services and see whether your services are reliable or not? So these are some of the advanced use cases of chaos that people are seeing, but you always start with some simple use cases such as spot deletes, and then you know go further from there.

A

So with that, let me give back to uh cortic. uh You know to talk about the road map and then the how to how to contribute to lateness.

B

Yeah, I think some of the roadmap that you're going to see now is driven by all these new use cases that tomorrow spoke about. So obviously we are looking at a hybrid set of targets. The the kubernetes targets, of course, were the ones that were initially added. We are looking at more experiments that can also target infrastructure resources, better ux for the chaos workflow management.

B

By this we mean how can you construct a workflow in a very seamless way and execute it and visualize it improved operations, support around chaos, engineering. Whenever you execute a chaos scenario, there are a few things around it. Where do you want your results to go? Do you want an artifact sync that that you want configured, maybe you're, running a load test along with a chaos experiment as part of your chaos experiment, along with your faults, so you want to see how the results of that benchmark really came, so you want to push it somewhere.

B

How do you ensure that the users on the platform have the right set of permissions and they are restricted in what kind of actions or faults they can do on your system? That's the are back and probes. Here is the ability to do advanced hypothesis validation.

B

So there are a few probes today, the scheme of which is getting continuously refined and new probe types are getting added and, of course, integrations with scenes if other cncf projects in the areas of observability at ci cdn just to achieve the overall chaos agenda, um contributing to litmus I'll, probably let ud pitch in. We have various areas that probably need the contribution so I'll. Just let him talk through that for a few minutes.

C

Yeah, you can contribute on a different different areas in litmus. You can contribute a whole experiment itself and there are other components we'll shortly discuss about that. So what do you need to have uh before contributing to litmus? You.

D

C

An id8 you need to have a chaos injection logic. um What pre and post cure checks you are going to perform, and how will you regain this stage after the chaos is completed?

C

Then the sex second step comes is filling the attributes file. You will have some attributes file that we need to fill to run an experiment to prepare to prepare code completion of the experiment.

C

Now, when you fill the attribute file, you will generate experiment, scaffolding that will have all the functions that are required to inject the chaos to perform some pre and post chaos checks. Once this scaffolding is done, you will have a business logic. You will have some business logic. Let's suppose you are running some part delete chaos. Then your business logic is deleting a pod. So that means this logic. You need to integrate with the functions after that. The next step comes is testing in the job.

C

Now, once you have all these business logic created, you can just dockerize it. You can run some jobs and validate whether your what your hypothesis is getting uh successful or not, you are getting the power deleted and recovered, or not.

C

Once everything is done now, you will provide some metadata to the experiments that will tell you about uh what the experiment does some more detailing and then you can simply add it to um commit it to chaos hub and it will be publicly available to litmus chaos hub now. The other areas where you can contribute is some, as we discussed like new experiments, you can already contribute now you will have you. Can you can improve faults? It must today have more than 50 chaos experiments.

C

They are all having different different for different features. They have different tunables. You can always go and add some few features in that you will find all these in the issues. Good first issues are already created there. You will have some ui ux bugs that you can fix, or you feel something that you can add.

D

C

You can go ahead and add. There are a lot of e2e test cases that you can write. As of today we have pipelines associated for each of the experiments. Some unit test cases you that you can add for the code. You will have the helm, charts, helm, charts are there for litmus installation, you can add some test cases around. It.

C

You will, you can add some documentations, you see. If something is not working, you can just go ahead and add the documentation fixes for that. You will have some. You can add your own use cases. You have some observability.

D

Integrations you can.

C

Add sim c you have some ci cd integration. You can add. So all these are the other fields.

D

Then experiment.

C

That you can basically use now uh to uh to contact our developers to discuss different topics. You can just log into litmus dev channel slack channel. It is present in kubernetes workspace and you can also join us on litmus github issues and you can see all the issues which you want to address there are. You can also share your feedbacks. That's very useful and raise issues raise issues for discussion for all these queries.

C

We are also available in slack. As I mentioned, we are on community sync ups. You can come and have your ideas there. We are also available in office hours these all. You can join these meetings. If you have some demos, if you want to contribute, you want to talk more about it. Please join and share your experience there.

C

With this. I would like to give the mic to sam for giving a use case.

D

Yeah, so so, basically uh now we know how and what litmus does what are some of its benefits and how you can use it, but let's see how we at cevo kind of uses litmus and what are our plans to even expand it further?

D

So uh cbo is a kubernetes provider, kubernetes service provider based on k3s, so we provide managed kubernetes, which is based on k3s, and how do we do that? We have our super cluster and our super cluster is ha is having kubernetes the k8s as a base. On top of it, we have the cube work layer and on top of that cubert layer, we have the tenant clusters that basically you when you log into co when you create the cluster, so you will be having the tenant clusters.

D

So when we talk about the 10 clusters, the tenant clusters are having the complete control and they can run untrusted and unsecured workloads. So we don't have control over that, but at the supercluster level, for us it is important to check the resiliency of our microservices to check the performance of a microservices that we are running internally, for that we have litmus, uh which is installed.

D

So why kiosk engineering for ceo is the first question that I'm answering so it it becomes important for us to test all the resiliency of our infrastructure, of our microservices, which are running on our super cluster, and for that we have litmus and now how do we do it and how do we do it continuously, because it's very important that you keep running your chaos as well.

D

So we have chaos implemented on our staging environment and we have created workflows that we keep on running in a continuous manner and those workflows are like different types of chaos: experiments which are there node level, experiments, pod level, experiments dns service. If it goes down what what will be the impact, we have to target those, because we are a cloud platform and if anything goes wrong, we we need, to kind of um you know, think from the perspective.

D

If any bad actors or bad people would be attacking, then it would be good for us if we can find the issues beforehand and fix it rather than wait for anything to happen, and then we investigate what is happening so we can create multiple uh chaos like karthik and uh uma and udit they mentioned they can be like bring your chaos, experiments, multiple uh chaos, scenarios that we can create uh so that we can think from those perspective, what maximum bat we can do to our super cluster and we can save it from anything that will be happening in production.

D

uh The next point that I want to cover is minimizing the blast radius. That is very important when you are doing chaos engineering, uh because as a cloud platform, we cannot, uh you know, just kill, anybody's pod or kill anybody's cluster. It has to be resilient, so we run it as of now in our staging environment, which is a complete replica of our production environment in a continuous manner. What we want to want to achieve next is uh to run it continuously, like it already has the ci cd integrations and stuff like that.

D

So in order like a new feature, is developed, uh so it will run all the existing workflows which are there and if all the workflows are completed successfully and they they like passes all the checks, then that particular feature is pushed to production. So I think that that continuous manner and moving from staging for to production using the chaos experiments would complete the story.

D

Now uh now I talked like this is a production super cluster and this is the tenant cluster, which is there and litmus can be installed. So how how this is happening so on staging? We run network chaos, podcast, latency and latency, and core services. Chaos on. We also give something to the customers. So, like I told you, we customers have full control over their clusters, but we have something called marketplace and in marketplace we have litmus from the maintainers, so it is a maintainer maintained application.

D

That is there that a user can install on their cluster and perform similar experiments.

D

So we use on our super cluster and we give our users as well the litmus to be installed on their kubernetes clusters, which is running on our platform to perform chaos, experiments on their cluster so that they can also um do the same pattern of in a continuous manner uh make sure their services and their workloads are running fine, resilient and yep. I think that's!

D

Those are some of the areas um that I wanted to cover like why chaos engineering is important and how to kind of use it and what would be to minimize the blast radius and how we give it to our customers. So that's what we do at sibo with that. I think I would give to uma.

A

D

A

Good, that's all we wanted to share in this brief time of 30 minutes. So we are definitely expecting more feedback to come in chaos is, is becoming commonplace in devops, so feel free to take a look at litmus start slow. You know, move fast once you're convinced that chaos actually is going to help you and we are always available to take more feedback from communities. So if there are any questions that we can answer here, we'll be happy to.

A

Yeah, as we say, we give license to chaos right, so you can. You can now use litmus and then play around and see, don't believe what somebody else is just see it for yourself break something and see you know you get a call from somebody else or not.

A

Well, thank you. We are available on the litmus channel litmus dev channel for contributors, so we are looking forward to announcing some of you there. Thank you.