Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2018 (Copenhagen), 14 Nov 2015

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Chaos Engineering WG Intro – Sylvain Hellegouarch, ChaosIQ (Any Skill Level)

Description

Want to view more sessions and keep the conversations going? Join us for KubeCon + CloudNativeCon North America in Seattle, December 11 - 13, 2018 (http://bit.ly/KCCNCNA18) or in Shanghai, November 14-15 (http://bit.ly/kccncchina18)

Chaos Engineering WG Intro – Sylvain Hellegouarch, ChaosIQ (Any Skill Level)

Propelled by years of success at Netflix, Chaos Engineering has recently seen a quick uptake as a critical discipline, for engineers and organizations, to keep a strong level of familiarity of production systems, as these are growing in complexity.Characteristics of Cloud Native systems (resilient, dynamic, isolation of concerns, automation, observability) lend themselves naturally to the chaos engineering discipline. This session is about introducing a new CNCF WG to achieve a strong community of practices, and potential API, that should benefit users when they want to apply Chaos Engineering.

About Sylvain
Sylvain Hellegouarch is the CTO and lead engineer at ChaosIQ. He has been passionnate about Open Source for as long as he can remember working on various successful Python projects (CherryPy, WebSocket 4 Python and more recently, the Chaos Toolkit). He has participated to IETF working groups such as the WebSocket (RFC 6455) and Atom Publishing protocols (RFC 4287). He has worked on companies of various size and loves a great team to learn from.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.

A

Welcome everyone. Thank you very much for coming and assisting to this presentation. I must admit. I was a bit afraid actually having so many people today, but when I see the massive room this morning, I'm feeling more confident I wouldn't want to be on that stage. um Let me set the expectation for for this session, because I have no idea what what exactly you're going to come and seek for I'm going to talk about a chaos, engineering working group for the CNC F.

A

Now, maybe some of you will you know I've come here to actually understand how to use chaos. Engineering, I, don't think that would be exactly the right session for that. So I'd rather set the expectation I'm going to discuss basically roughly the idea of a chaos engineering and then why we need a working group. Why I believe we need a working group on that, but because the idea of a working group is to collaborate- and all you know federated on the idea- you'll be more than welcome to participate the discussion after that.

A

So I'll I really hope that we'll you'll get what you you wanted to get from a decision right that works. Let's have a clue. No come on, see defeated. First slide, that's chaos and rank for you.

A

That's fine I'll do with all right, I love IQ for for you on chaos, engineering, so I, don't know. If you have aware of IQ those are just short sentences, I try to which you know bring you large. You know our stories in few. In few words, so we love our systems yeah. Thank you, yeah that works. Thank you very much, but I tend to fail all the time now. Finally, I'm sure some of you think that's not true. Your system never failed.

A

Well, they actually do or the system underneath do fail, but we tend to turn a blind eye as developer. You know we we not really teach at all to actually think about failures. We are taught to build things. You know, deploy them and then move on to the next feature, but that thing that we've put in production doesn't stop there. It will fail.

A

It will break in some fashion purely because either you have a bug fellows on its own or the thing underneath will break or your users will break it, but we turned to you know. We tend to think that you know it's a bug. I, don't want to deal with that bug. I found that extremely interesting to actually look at wide field and actually learn from that.

A

So I'm I'm going to start with a story because you had lunch, you may be bored with what I'm, seeing already so I'll try to catch up with a story. That's a story of the man in his phone based on true events. Right, so no shenanigans, we CSI it's just the strip's recently, you know Matt Damon's I am a man text security seriously, yes, set up to offend all his accounts, who's using 2ff for their Gmail, or you know things like that here, right, pretty much everyone, or at least a large.

A

You know, group of you it's using Google Authenticator now I was using that one to actually get the code unless so, but already good good. You know good portion, that's a story! Serious story now a lesson learnt that's a first stage of a story: the phone dies.

A

So what happens if Google I think Authenticator is on the phone that I stress died? How do you get the codes? Well, forty battery, by the way, you probably never predicted that did you oops? The man says because it's for all audience I said oops, but usually those worlds are way. Different I can't produce the two FA codes anymore and obviously because you know I'm an engineer and I'm I know better I didn't printer.

A

The codes died, whose printing the codes that they give you when you set up the two FA and put them aside on some of you, Bravo.

A

So a recent is learned one day: I shall not never forget, don't bet on a single device for your Twitter at least don't mix o2. Where you come, you know you bring, though your your codes from and where you using them shocking events who cure new phone gets stolen. The guy. You know the man has you know? No luck. Perhaps, but that's that's a story. We have thing here.

A

Hartman says I'm using my laptop to generate codes I'm, not using the Authenticator anymore. The Google one I learned my lesson. Prime, is the phone was torn in a foreign country and the laptop is nowhere to be seen so I can't access the codes. Well, I had a man contacts Assad who again you'd, get to Jesus, probably different worlds, one spot the phone was stolen. So if the you know the thief gets into the phone all emails, everything is actually freely accessible because it's already Ascenta gated.

A

So now you can actually exclude the person from using your phone and you come generate another code because well you don't have your laptop you're in a different country and again the man you better, but didn't do it because you know he's an engineer or some sort. So it's still, you know in bad luck and you understand Islam print, stupid codes, but then again to be continued, because if you get the code on paper, do you bring them I mean gets. You know it gets a bit crazy that way.

A

Now what the point of that sophisticated story that everything you know surprises come you know that right and if you get familiar with what you know with those surprises or unexpected events that will save your bacon. So basically, what I'm saying here is for that. That's very what you would want to do now is tomorrow. Ask yourself what if my phone is stolen or dies, can I still access my different accounts? What happens to me? Are the user? Well, that's exactly what you would want as a case engineer to think of your system.

A

What happens if this happens? That thing that you probably don't think it would ever happen actually does happen. So that's a mindset as a casing engineer. You probably want to do to have what happens if, for the credits, I am the man, so it happens to me. My phone died six months later got stolen and you know, and I still don't know what to do now with the print with the codes.

A

Perhaps two phones, I, don't know all right. Let's get started now because that story was fun, I guess, but this is what we have for we're going to start with. The principle of cows Achilles, who has who is familiar with those principle, actually come back to that who's, familiar with cows, engineering in general, right who's, practicing a variation of chaos and during like resiliency testing, some people already so good because, like I was saying, some of you are coming for new knowledge entirely.

A

Some of you already know quite a bit so I'm trying to find the right balance here, but let's start on a principle of chaos who has read them or gone through the page. Perhaps right a few of you, it's just one manifest release. It's quick to read it's interesting to see what comes from so I. You should really look for it and just read the page, but basically the the first sentence is: is this one? It's a discipline of experimenting on distributed system to build confidence when you know bad things happen, and it will happen.

A

So it's really a when not and if now the question here again, we are about to talk about a potential for working group at the scene CF, so that definition as good as it is will be. You know, discussed by the community, maybe will drop the distributed. Because do you need distributed system to actually do chaos engine? Do you need you know, communities to actually qualify to do Kuban, chaos and ring? Probably not it happens if you have a production system? Basically, that's if you have a production system, you probably benefit from that.

A

Now, that's a quote: I took from I know the once you get the slide go to that URL. That PDF is very short, is two pages PDF to read it's a very interesting one.

A

About twenty years ago, something- and it's all the cases for failure- it's really interesting to read, but one that really struck me was the more you get familiar familiar with some with asset and basically, you know danger in your system the more the better you get at fixing or ending that who's been on call sometime and thinking god, please don't let them the system break down, because I have no clue a few of you right, I've been there. You know where you like come on, come on, come on because really you haven't practiced.

A

Your colleague is an experiment that is always save your ass or you know sorry for well. You know what I mean so I get carried away, but he's not here today, maybe isn't already. Maybe it's just you know it's your turn to be on call or whatever, but you haven't practiced you may have given you, you know wiki, maybe I've, given you a lot of things, but you haven't practiced you you're not familiar with with the problem. You're going to get slow, stressed and stress under failure is not good. It's not the position.

A

You want to be in so practice before and whiskerson drawing so that when that what happens, maybe a different slightly different, you know scenario. At least you feel confident you can do it. You know maybe the venue's, who took to talk to anything like that, so you're, more practice, failure the better you get at them.

A

So some contexts, Netflix is the origin for that now system. Failure system, theory, is not, you know, invented by Netflix, but the coin. The chaos, engineering term and they've done a lot of things in our industry in general micro services and plenty of those things seem to be born there. So usually we try to pay attention in about ten years ago, I started to explore about the ideas of failure in the system.

A

So at first it was just you know, inside the system, then the release chaos monkey, which was certainly some of like an outcome of that from the system from from the expiration it's using kills monkey, maybe also just tried alright, okay good.

A

Then they started to organize himself around the role of chaos enduring by hiring properly kills engineers and now they're. On a dig, but not Netflix per se, but people from that era, educating with the principle of chaos or the kill, centering shot book that you can find for free on O'reilly and so what's next, basically, because Netflix seems to have done, you know over ten years you know covered all the ground. Well, it's our turn. Now it's all world that that diagram is actually not up to date, the up and don't try to read it.

A

It's way too small. But the idea here is to just show the sheer number of people and variety of people already either doing it or you know creating tools around it or talking about it. So it's really become a whole ecosystem or on chaos. Engineering. Even if it's quite new, it's it's it's it's fast. You know on the speed line. I think the chaos in joining slack is about eight hundred people right now.

A

So now quite a few people interesting and subject so why I was enjoying cost of ignoring witnesses, existence not ignoring witnesses but ignoring that they exist as different. Here you may know that a weakness exists, but you've met the choice. To ignore it. That's fine! You may think that from a business perspective fixing it is not actually that important and that's okay, but ignoring that you don't know, but weaknesses in your system is a danger for your business. So it's going to exceed the cost of actually going finding them.

A

It's a bit less the same principle with testing, basically but broad production. If it, you know it's it's very closely. If you find something you know if, if your user basically run into an issue in production, not just in in hard money, but the the trust they get into your system is listen and you don't want that because it's it's a cost difficult to actually gauge, but it's going to be a lasting long effect.

A

So we have pitfalls everywhere. System grow in complexity, so fast I, guess you know that and if the funny thing is we seem to build new technologies that create new problems, trying to fix all problems but actually creating new problems. Discipline scales, injuring is not much different in the sense where it's trying to fix something that may not have existed in the past.

A

So what what can I'm sure you find yourself? You know in some some of those those points. We have lots of moving parts, especially if you want micro-services, you know kind of architecture. You know things are moving fast up and down all the time. You've got anti-dependence CC between the season. You know between the services, but outside I mean every day now you're going to use slack you're going to use get up once I was I, wanted to actually do release. So I pushed my code to get up and try.

A

This was supposed to build, but Travis was down so I couldn't actually even build and then even deliver anything. So it goes free. You know early, sometimes sick, get up that is done. You can't even push the problem is what, if the push you want to make is the push that matters to your system? So really you want to know that all those those links are weak points in your system. We change it at first. You know we change out the first base. I guess you know that fake security is mad major component.

A

Now we can't just leave that to someone you know somewhere or trusting just oh I'm using TLS I am a secure good. You know security guru its. It goes a bit beyond that. I guess: structural regulation, who's suffering from having to deal with gdpr, I'm sure yeah, a few of you right, no matter the the point of the the regulation. It may not may not be a good regulation, it's imposed on you and you didn't. You know, look for it!

A

Reliance on peace, we don't control weaknesses, ripple effects, especially even more in distributed computing like and and communities and the blebbing culture. It's not really a pitfall, but I think it is in some fashion. We point fingers sometime in some teams. I'm, not I, think we're getting better as an industrial for that, but it still happens where we're going to blame the one person that press the wrong button. When, basically, we should ask why that button could be accessed by that person.

A

What was in the system wrong and what we should fix in the system, not blame that person so benefits of chaos, engineering, because we become proactive, we increase our confidence in our systems and we increase our trust in our teams. Now, I make a distinction between confidence and trust. Now, perhaps it's because I'm English is not my nature language, but to make confidence it's something we built by seeing so you get. You know, monitoring that tells you. This is happening all the things.

A

You're, probing your system and you get you know getting on data and data and data. That's building your confidence that something is happening. Maybe automation is going to build your confidence. Trust is where different trust is a belief that you have. That is somewhat subjective, that that person is going to do the right thing or that thing is going to act. The way you want, but you really want to have both it's not because you're using G cloud for Apollo, AWS or anything or anything.

A

You know you're, confident, they're doing the right thing right, but until you've really experimented, you probably don't trust the system. Yet so having both scale, centering is going to actually provoke the systems and provoke your teams and you're going to build better confidence and trust in in both both of them. So, but why? Because it proves your system, availability and safety. A safety is important here, because you want your users want to be safe. All kind of you know way of being safe in their system.

A

Your data, their data and, and you know in self-driving cars, I guess safety is quite you know, quite a key factor. I guess, availability. Sometimes we hear about resiliency as engineers we care for a zillion, see your users care for your system to be there. They don't really care how you do it. If you manually you pressing the button beyond the scene, they don't really care water care.

A

Is that system is there and that will lead to a paean dressing users which, as engineers, that's where we are therefore so really kills enjoying is going to give you a better production happier? You know, as an engineering team, better knowledge of your system, not just you know, I've deployed now I'm. Turning back to my to my desk, but how it runs, you know, that's good thing. Annual users are going to be happy about that. So why is sincere working group?

A

It's a natural encounter, I! Think since EF and we've seen that this morning, during the keynotes as CN CF, has led solid Phoenicians for community and distribute industry collaboration. Really, when I look at Cuban t's, it's actually impressive. What happened? You have all those players coming together and agreeing on one platform. So what that means is you, as a user? You know have confidence that you can use those various players and it's always the same. You don't have to relearn something you don't really have to it's a same community, so all the benefits of learning.

A

Something is doing even if you move cloud you're going to carry that knowledge across. So that's really good and that's actually the most powerful thing, I think from the scenes here from from communities and chaos enjoying is also sociopath. Socio-Technical decision yeah!

A

A

Well. It could be seen that sorry for that and it's a social, technical discipling where you learn from experimental corporation. It's interesting the experimental corporation here because you cooperate from different teams. Let's say you have a big organization, you have a service support. You have, you know various you know kind of teams there, your support engineers vample, could create chaos and doing experiments directly from what users experiment in in general and that feedback to your user to your developers.

A

Who could actually run that experiment and say well, that's interesting because I didn't we didn't think of that. That's a real production use case and and having that under the scene, CF means all the knowledge that we all have in production system. We can start gathering that and may be discussing how to push the boundaries of what we the way we test them.

A

If you will and I think it makes sense to have a working group on ciencia from that now don't worth what are the goals and angles of a working group here, let's start with non goals we are since EF is not a standard body, it's not the IETF, we're not going create an RFC this. We is just really aware of federating different. You know inputs and people and get interest in there, so we will not try to bless one project.

A

What that means is one vendor come shouldn't be blessed just because that's the biggest vendor of ample there's, not the point of the scene CF or no of the working group. Now what are the goals to figure it industry and the community at large and create educational material? That would be already awesome to have. Actually it you know, educate those who want to actually explore their system life either staging or even production propose an industry-wide effing ition that users can make their own back to the principle of care.

A

So definition the give is right, but is it the right one? It's a good one, but is it the right one? We have to decide that, as together provide a white paper, it seems that most working groups are run run. The field create work, white paper, I think that's a good thing, because at least that gives you you know a good summary of all the discussion and where they led to and maybe crowd the CN CF landscape even more. If we can.

A

That's probably difficult seeing a big, it is already discuss the ethics of doing chaos. Engineering chaos centering is not about breaking stuff. Without you know users, you know knowing about it at least when I say users, your teammates. So there is an ethical question about how far you go about breaking something or you know, harming your system. I. Think it's interesting to to get there.

A

That's that's why it's different from you know some security aspect where they they do that. You know in the shadow, if you will, which is in in interesting itself, while still enjoying, if is very much in the transparent public approach, that all the teams should know about at Netflix, a try not to do for ample kills and during during weekends, or you know, out of working hours wouldn't be fair and finally discuss and potentially come up with an IEP is specification. Now you might wonder what would look?

A

You know an API would look like and I'm, not sure we can come up with one, but what is it sure is all the cloud providers all the system. You know that you're using they have expertise in there and if they could expose that expertise as a red button that you could press in an automation somehow to actually stress their system to see how your application is doing, then that would be interesting. So whether or not we can come with an API I, don't know, but that's sort of discussion is, is powerful.

A

Next steps, listen to the CN CF community. That's probably a good first step, listen to people, so we have well. We have. There is a slack channel on the cnc of slack called calcium sharing. You've got another and we see the link beef. After that you have a bigger slack, which is dedicated to kids, enjoying that you know both work.

A

We want to propose the working group right, not it doesn't exist, we have to go through the you know the process of submitting that we walk to the TOC and that if they agree that becomes a real working group- and you know everything that comes with it.

A

If you want to join the discussion, so obviously you can't clink-clink those links right now, but basically, when you get the slides the first one is, we started with just a document where we started to teach at now. Google Docs is good, but you can't comment at you know so many people. So it's probably for you to ever read the discussion that already happened there and you know comment if you need, but it's not really the best place right now. It will move on from that.

A

The first meeting should happen like within one or two weeks, so if you're interested in it will be some unzoom or something like that, and then you know come to the public, the public forums, the first one is a Google Groups forum, so it's male based and the second one is a big biggest slack community that you can find on chaos and rain and yeah to finish really I.

A

Think if we come together and we learn from each other, because the funny thing is when I go and talk about catering to people at first I may be may, but quickly they realize that yeah. We have production incidents. Oh that happens. Oh did I tell you about that. The funny thing is, we have become very good at as developers together exchange no see how to do better. Like all those things around you know development, but as herbs there is a lot of you know, grant to cover still and care.

A

Centering is really an interesting one because it says don't wait for something to happen, but actually provoke that. So, if someone, you know, is using a system similar to you and as at the incidence it's interesting to learn from it, obviously, but then maybe see if you can actually replay that in your system and see how your system would behave. An example that you may have had I've used that yesterday, let's say your certificates expire, alright, who's as add you know a certificate that has expired in production right. Few of you right.

A

The question is not report. It's not really a problem that expires, but it turns miss two things as okay was injuring. First of all, did you get? Did you have an alert for that? That would be a first experiment to actually set an exit. You know a certificate that is bound to expire within the next five minutes and see if you get an alert for that, then you've got something already. You don't know that you it's going to expire and, second of all, really the prom is. What is the impact?

A

What are the ripple effects of that certificates going? You know crazy from the outside world as browsers are very going becoming. You know, strict about that, and your users won't see the application anymore. Probably from the inside of the your application, probably services will start having difficulties to actually even communicate. I'll give you an example: I have heard of a company was, she is using Microsoft and LDAP on. You know: Active Directory, all the users were already connected.

A

Most of them keep the code session open for a long time, and then someone was hired and couldn't actually again. They couldn't understand why and I started trace all the network and everything and I realize a few days before that patched that set up a patch on Active Directory server from Microsoft, but that has changed the TLS. You know the device schemes you can use on TLS and suddenly their clients could actually connect because was using a wrong TLS version, but already connected. You know.

A

Users didn't see that so I had a kind of prediction issue that you want probably to discuss together and that's why the TF is interesting and then see if we can actually come up with scale, centering experiments and we can exchange and share together and say, try that and see, if that, actually you do something to your system right, I. Think! That's all for me. Thank you very much for listening now. I, don't know what time it is and how much I have left. I've got a few minutes to spare for questions.

A

So, if you're interested you know raise your hands might be too and just quickly on Friday I've got another session where I'll do some demo about chaos and drone using the kills toolkit. So, if you're interested in seeing that we can come back on Friday morning, if you're still there right anyone question I'll, be happy to answer question more private if you feel more confident after that, anyway, all right well, thank you very much. Then.