Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2018 (Copenhagen), 14 Nov 2015

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Lightning Talk: Chaos Engineering In Practice - Paul Jones, Capgemini UK (Intermediate Skill Level)

Description

Want to view more sessions and keep the conversations going? Join us for KubeCon + CloudNativeCon North America in Seattle, December 11 - 13, 2018 (http://bit.ly/KCCNCNA18) or in Shanghai, November 14-15 (http://bit.ly/kccncchina18).

Lightning Talk: Chaos Engineering In Practice - Paul Jones, Capgemini UK (Intermediate Skill Level)

Chaos Engineering facilitates experimental fault injection into our systems and infrastructure to identify systemic weaknesses and ensure resilience and confidence in our systems. We will discuss the principles of Chaos Engineering, as well as practical examples of how open source tools can enable us to cause, learn and benefit from the ensuing chaos, and how we can use automation to continuously drive orchestration and analysis.

About Paul
Paul is a Senior Software Engineer with Capgemini, with experience in cloud native tooling, container orchestration and infrastructure as code deploying AWS and hybrid clouds.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.

A

Coming everyone pleasure to be here today today: I want to cover some open source, open source chaos, engineering tools to facilitate fault injection and influence and chaos, engineering principles into our environments. So simply put we want to run disciplined chaotic experiments to identify weak points in our system and fix them before they become problems. So caerse engineering is fundamentally orientated around creating resilience in our applications. So hopefully these tools can actually help us influent that today, so failure is inevitable will will we need to be ready for when it occurs.

A

So this manifests really in a chaos engineering perspective in three stages, so first we want to hypothesize. So how can our system fail? We don't want to go around randomly killing things. We want to actually have a control experiment, some metrics. We can capture to actually prove that this is occurring and hypothesize. How that will affect the overall system. We then want to contain the blast radius, so we want to test the smallest scope that will give us the maximum value out of the experiment and then, after that, we want to scale or squash.

A

So you want to fix the issue automate if possible, so we can actually cause continuous chaos and then increase the blast blast radius until we're at full scale.

A

So first up, we've got cute monkey, so cute monkey is an implementation of Netflix's chaos monkey, but for communities clusters which will randomly delete key methi's pods in the cluster I. So cute monkey works as an opt-in model. So we only terminate scheduled terminations for communities apps that are explicitly stated as such in their deployment Hamill's.

A

So here we've just got an example configuration for cute monkey itself, so we create a config map just stating what namespaces you want to blacklist and whitelist, and some scheduling- and here we've simply took us- just got the cute monkey deployments and then from a victim perspective. We've got a normal deployment and we just add some labels onto there. Just to say this is a victim enabled then we're just setting some of the data around mean times between failures and what mode we want to kill the application in next up.

A

We've got cows monkey for spring boo. So this is a small library which you can integrate as an independent as a dependency in your existing spring boot. Okay, application. It leverages aspect-oriented programming, so you don't actually have to change any of your source code if spring break hairs. Monkey is on your class path and activated the chaos monkey profile name. It automatically scan your application for components annotated with these spring annotations controller service repository that kind of thing. There are three potential attacks. We can inject so there's latency exceptions and app killing.

A

So, as we can see here, it's just an actually dependency in our pom and we have some extra properties in our application properties and in this scenario, which is enabling the cows monkey and we're injecting latency on to controllers. So if I just jump over to this, so here we just have a standard spring boot application. We also have a controller set up with the right annotation and just creating the hello endpoint.

A

If we drill into the aspect and on this method we see here, we've just got the standard point cuts for controllers and public methods. We drill further into this and we call cows monkey and here we're just picking out the properties from the application properties file. We stipulated so we're just picking up the fact that we enabled latency, if we drill in further we're just artificially injecting some latency here, just through some sleep cool and that's what we just saw.

A

Lastly, we've just got powerful seal so again, this is test and resiliency of the applications in communities clusters. It comes in two flavors, so you've got interactive, and so we can literally just run that command there and that'll pop a shell using our cube config, and this will call the cloud API and the cube API. So we can stop start delete nodes deployment pods. That kind of thing, if you want something a bit more, automated we've got autonomous modes. So here we have a policy yeah mall, so we're just stipulating some matches for the nodes.

A

So what names IP? That kind of thing some scheduling we only want to sort of course, chaos during business hours, for example, and then the actions to take how we want to terminate those nodes and for similarly for pods. So we've just got some selectors here, so we want to match on namespaces and labels and names we can filter further on these and what state they are actually in and then we've got similar criteria in terms of scheduling and how to kill the applications cool.

A

So just to take away from that, so it's disrepute systems, they've grown much more complex failures have been much more difficult to predict, so we need to be more proactive around how we actually learn from our failures. Chaos. Engineering is accessible and simple to set up as we've just seen, with minimal changes to our deployments and our source code, so be a chaos champion in your organization and thanks for listening and have a great confidence.