Kong Kongcast: The API Connectivity Series, 27 Dec 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Embracing Failure With Chaos Engineering With Jason Yee from Gremlin | Kongcast Episode 7

Description

In this Kongcast episode, Jason Yee, director of advocacy at Gremlin, explains the concept of chaos engineering, why even the best engineers can’t control everything and tools and tactics to help build app resiliency.

Learn more about Gremlin at gremlin.com/community or get certified at gremlin.com/certification

Subscribe to get email alerts for the latest new episodes and see which guests are coming on the show next: https://bit.ly/3zQV71e

▬▬▬▬▬▬ TIMECODES ▬▬▬▬▬▬
0:00 Kongcast Show Intro
0:28 Welcome, Jason!
1:20 What’s Chaos Engineering?
10:30 Chas Engineering Patterns
12:57 Phases to Use Chaos Engineering
15:10 It’s Not Just About Breaking Things
18:57 How Does Gremlin Chaos Engineering Work?
17:16 Demo: Gremlin Chaos Engineering
33:47 Book Recommendations from Jason
35:27 Thanks for Joining, Jason!
35:38 Kongcast Show Outro

▬▬▬▬▬▬ ABOUT KONGCAST ▬▬▬▬▬▬
Hosted by @ViktorGamov and Kaitlyn Barnard, we interview software developers and technology leaders at the top of their game every other week. We’ll also give you the tools, tactics and strategies you need to take your distributed architectures to the next level. Kongcast goes beyond the buzzwords and dissects real-life applications and success stories so that you can tackle your biggest connectivity challenges.

▬▬▬▬▬▬ KONG COMMUNITY ▬▬▬▬▬▬
⭐ https://github.com/Kong/kong
🐦 https://twitter.com/thekonginc
🌎 https://konghq.com/community/
🍻 https://www.meetup.com/pro/kong
❓ ️https://discuss.konghq.com/
💯 https://konghq.com/kong-champions/

#ChaosEngineering #Gremlin #ChaosMonkey #Kongcast

A

Kongcast-Jason-Yee-Gremlin-Youtube.Mp4 hey, everyone,, I'm kaitlin barnard.

B

And victor gamov.

A

In this series,, we chat with software developers and technology leaders to tackle your biggest.

A

Api connectivity, challenges.

B

Stay tuned to this episode for tools, tactics, strategies that will help you to take your.

B

Distributed architectures to the next level., let's begin.

B

And today,, I am extremely excited to welcome jason yee, director of developer advocacy.

B

At gremlin., hey jason,, it's great to have you on the show.

B

Hey, victor., it's really exciting topic and I'm looking forward to our conversation about the chaos.

B

Engineering., but before we start and jump right, in. jason,, can you tell about a little.

B

Bit about yourself?, what do you do at the gremlin specifically? and like how you end.

B

Up in this field of chaos, engineering?.

C

So I'm I work at gremlin as the director of advocacy,, which means I lead our developer.

C

Relations team, also including events., and as with any small startup,. If you haven't.

C

Worked in a startup,, it's all about wearing multiple hats., so I do a bit of customer success.

C

As well,, so I help out our customers and manage a few accounts and lend a hand wherever I.

C

Can. so lots of different things at gremlin,, but chaos engineering is really why I joined gremlin,.

C

And it's something that I'm super excited and passionate about.. My experience with chaos.

C

Engineering started a long time, ago, actually., I think most people who have heard of chaos.

C

Engineering are familiar with things like netflix's chaos, monkey,, which is coming up.

C

On its, I think, over 10 years old now., it's pretty old.

B

C

Yeah! time definitely flies, and at the same time, it feels like it's still a new thing,.

C

Right?, so we have this weird, maybe temporal vortex within tech, where old things seem new.

C

Despite being old,, or maybe it's just that they get reinvented., and I think that that's.

C

True for chaos: engineering,, it's been reinvented., so I heard about it back when I worked at.

C

O'reilly media and I was helping run the velocity conference after that and went to datadog.

C

And at datadog, we actually did chaos engineering,, so I was able to really get my feet wet or.

C

Get my hands dirty, as you will,, and help with that practice. and then that's how I.

C

Met the folks at gremlin,, and so I joined the team there.

C

The history of it, though, is basically, as we talk about reinventing some of those early.

C

Folks from netflix and amazon who pioneered chaos, engineering realized that not everybody.

C

Is netflix or amazon., and so they decided to make a platform that would make it easy.

C

For everybody else to do chaos, engineering., and so that's, basically, what gremlin is.

B

That's exciting. and how how I came about to this topic is,, like many, I would say, like.

B

Eighty five percent of the people in the world, they may be reading the netflix engineering.

B

Blog and during the time of the great migration to cloud that netflix is famous about,, we.

B

Learn a lot of cool tools that they do,, starting from the tools for their build process. That.

B

Will end up being open source as a kind of like a nebulous set of plug ins for for, for.

B

Gretel and some of the cloud management tools that end up being the kind of foundation for.

B

Spring cloud world and the obviously the chaos monkey. and when I was preparing to do this.

B

Episode, I was finding the very interesting youtube channel, and the host of this youtube.

B

Channel have some resemblance with you., it's called reliability, engineering, and the host.

B

Was talking about the chaos monkey. so what?? What is the chaos monkey like?? Why it is important.

B

Like what?, what's what's all about? you know, during my times,, we don't need to have a monkey.

B

We might have a night janitor that will be just randomly disconnect the servers, because.

B

He came into the office and starts like mopping the floors and there was a, you know,. The.

B

Cables connected to electricity. and after that, all of a sudden, he disconnected this.

B

This is how we're doing the the chaos engineering back in the day.. What changed?.

B

Oh yeah. yeah,, I mean,. I completely resonate with that. the pulling the cables actually.

C

One of my first experiences out of college and this was,, you know, several decades, ago,.

C

I was testing out whether our ups system would work right.. We we had a bunch of servers.

C

It was obviously on prem because the cloud didn't exist., and so it's like we have this.

C

Great battery backup system., let's ensure that our servers stay up and pull the cable.

C

Out of the wall., except for I pulled the wrong cable., I pulled the cable from the server.

C

Into the ups,, so obviously that didn't work.

C

Yeah., so another example of this is that you randomly knock off the the the your box of.

B

Your servers and the sound your servers died. another thing you can just like randomly start.

B

Taking out the disks out of the disk crate. and after that,, you see how your data will.

B

Be corrupted., but we did this unintentionally., like sometimes it was just like stupidity,.

B

Sometimes an inexperience thing.. My understanding of this approach is that, instead of and I'm.

B

Totally stealing your quote from you, one of your episodes on your youtube show,. Instead.

B

Of like a fighting and build one hundred percent resilient system,, we need to embrace this.

B

Chaos. we need to embrace these failures and understand that these things will happen.

B

And, rather than building 100 percent reliable systems,, we need to build the systems that.

B

That will be resilient to failures...like murphy's law, says, bad things happen.. You.

B

Know,, if something will happen,, it will happen.. So you be,, you need to be rather prepared.

B

For the way how the system will works. in the,, you know, breaking state, rather than,.

B

You know, fighting for one hundred percent up time,, something like that.

C

Yeah, absolutely. and so, you know,. If we think back to that original chaos, monkey,.

C

That was absolutely it was. How do we embrace failure?? How do we, how do you just choose.

C

That failure is normal? and so that original netflix team, their idea, was, well,. Let's.

C

Just introduce failure., and so it's always there,, and then our teams will work around.

C

That. they'll build systems that will respond, gracefully to servers going down and they'll.

C

Just restart., and so as a way to constantly test as the practices evolved, a lot of it.

C

Now is with cloud native systems. they're, so complex, you have microservices and you.

C

Have all these dependencies. and there's an understanding that, when failure happens,, it.

C

Sometimes or many times is not what you would expect. and so one of the ways to get better.

C

At your systems is not just embracing failure,, but it's understanding that you don't know.

C

Everything about how your systems will behave. and so investing time in that, investing time.

C

And really trying to dive deep and understand how they work in the real world and not so.

C

Much how they look on an architecture diagram and how they're supposed to work,, but really.

C

Introducing that failure and seeing what really happens in the real world.

B

And what kind of component the system is depend on and which components you need to have additional,.

B

Maybe,, you know, the backup or like additional set of replicas or things like that. and apparently.

B

This topic is still very relevant in the cloud world.. Would you agree with about this?.

C

Oh yeah. absolutely.

B

The vendors, kind of like a providing us some assurance, that, yeah, yeah,, we're going to.

B

Be running your infrastructure. and yeah, yeah, yeah,, we have a,, you know, the multiple.

B

Zones, we have a multiple regions and like we can provide you with some reliability.

B

But the things that happened, I mean, like five years ago when they or some of the tractor.

B

Literally just hit the amazon data center or not even data center itself., but like hit.

B

This electric station that was near this data center.. This is the example of the things.

B

That unpredictable- and this is where the the people's factor like human factors, comes into.

B

Play. yeah,, you can build the data centers. one hundred percent protected and things like.

B

But, you know, all of a sudden,, sometimes the the road, the accidents will happen,. Then.

B

Someone will crash into your data center. and be prepared for this type of scenario.

B

This is what I think the essence of the chaos engineering, right?.

C

Yeah,, the essence is that you can't control everything,, even if you're the world's best.

C

Engineer., you can't account for everything because, you know, again, in our cloud world,.

C

In our distributed, world, you're constantly interacting with other services,, especially.

C

Outside of your domain, right?, you use services like kong or you use services like gremlin.

C

Or whatever.- and these are outside of your control., and so you have to have a better.

C

Understanding of when things fail,, whether that's yours or these other, these other systems.

C

What does that actually look like?? How does that affect you? and we've gotten to a world.

C

With everything so connected that your customers don't care., if amazon goes down and it's not.

C

Actually, amazon, it happens to be some major internet provider that runs the backbone between,.

C

Say,, the u.s. and europe, nobody cares. everybody just thinks, hey, amazon's down and you take.

C

The blame for that., and so with your own systems,, it becomes that reliability domain is beyond.

C

Just your own systems, and you have to start accounting for all of those uncontrollable.

C

Factors. and so again,, a big part of that is understanding how your systems work.. I.

C

Don't think anybody would expect your systems to operate if the entire internet went down.

C

But at least you know,, if, if one of your dependencies or your services goes down,, you should have.

C

An understanding of what will happen.? What does that look like, so that you can respond.

C

Appropriately?.

B

Netflix also was known as the organization., apart from like advocating for this type of.

B

Things., they also were advocating for certain patterns, for building resilient applications.

B

To things like circuit breakers., how you can,, especially because we're talking about microservices.

B

World like in the modern world, we're not deploying the one monolithic application, because.

B

Like if one thing will go, down, everything will go down., but with the microservices world,.

B

Some of the components will continue to to run in the case of partial failure of some.

B

Other components. like, if you have an ordering system and you need to have information about.

B

Address and address is provided by another type of api, like how you would exist if this.

B

Service will go down., so you need to have either kind of some local state cache state.

B

That you can use this and, after that, invalidate as a recovery procedure and things like that.

B

So that's that's something that is that I also know as a the thing that we learn from.

B

The netflix experience. like, can you give some similar examples to like some of the.

B

Patterns that we start seeing more and more in the application, architectures and application.

B

Design? yeah, absolutely., you mentioned circuit breakers,, but that also usually goes along with retries,.

C

Right? a lot of times we have circuit breakers,, not because of external calls,, but because.

C

It's some sort of cascading failure of retries. and how do you tune those?? Do you have...what's.

C

Your back office look like or when something fails? is your system just going to start.

C

Keeping to retry and queuing more and more requests and building that?, and so soon you.

C

Have that thundering herd problem., so things like that are commonly tested with chaos, engineering.

C

A lot of things like retries are easy to test by simply introducing some latency into your.

C

System and beginning to dial that latency up and seeing at what point you start to hit.

C

Your retries and what that does., if it does build up into a queue, or if it handles that.

C

Gracefully and realizes that maybe that system,, although it's latent and it might be responding.

C

Slowly,, maybe we just consider that as actually being broken and down, and we kill that we.

C

Start up a new service and hope that that doesn't have latent performance.

B

In your opinion,, when the people start looking to this approach for chaos, engineering, should.

B

They look at this one as kind of things that will be running side by side with their production.

B

Thing, or they need to start a gradually and introduce it in the in a stage of integration.

B

Testing, for example., they the build the feature,. They would deploy some integration space and.

B

After that will have chaos testing as a part in other stage, as a, you know, before, like.

B

Acceptance test and before the things will be rolling into production. so.. To summarize.

B

It, in your opinion,, is it like a chaos, engineering, part of the production thing or it's a part.

B

Of like a pre-production integration, phase?.

C

I think it's both., so oftentimes people are very afraid to just go straight into production.

C

And I think again, that's a sort of a misnomer, or maybe an evolution from those original.

C

Netflix chaos, monkey practices., they just went straight in production and started killing.

C

Things., that is scary,. I mean,, if, if you're thinking of that and you're not scared by.

C

That, then maybe something's wrong., so I highly recommend that people start in staging environments.

C

That they work just like they would with code, right?. You build things in development,. You.

C

Push them to staging and q&a before you roll them out into production., and I take the same.

C

Approach with chaos, engineering that by the time you're pushing into production,, it should.

C

Just be like any of the other code that you're working on., you should have full confidence.

C

That it's going to do what you think it's going to do. that said,! When it comes to the.

C

Automation, we often find that people start manually and they'll start to test just with.

C

Manual, chaos, experiments in those pre, prod environments and then manually do them and.

C

Prod. and then from there, they'll, usually roll that into their their automation,. Their.

C

Ci/Cd pipeline for that testing. and so generally,, it's a manual process into production and.

C

Then automation to prevent regression., it's typically how I've seen it,, although lots.

C

Of companies work differently., so I have worked with a few customers that are just straight.

C

Into ci/cd. they know they want this,, they know exactly what they're going to do, and.

C

Everything has to be automated.

B

And this is very important thing to point out, I think, is that the chaos engineering.

B

Or like like a failure, injection,, it's not necessarily needs to be, you know, killing.

B

Things or like a break in services. jason, mentioned think about like introducing latency.

B

And see how your application will behave if there's a...it's a normal situation when the.

B

Database become slow because either there may be saturation of network, because someone.

B

Else, all of a sudden start using the same channel to your database., or maybe it becomes.

B

Slower because of the indexes in this database are not optimized and things like that., but.

B

When we develop certain feature,, you thinking under the condition that everything will be,.

B

You know,, nice and green and that it would be sunny in unicorns and things like that.

B

However,, just simply looking how your system will behave by increasing the latency between.

B

Components already would be like a big eye opener in terms of like how certain things.

B

Are configured in in your system or like maybe some of the requests that you think needs.

B

To be synchronous. and when you're testing this on unit test, or even maybe some integration.

B

Test,, you see the response. Time would be very quick. and I think, yeah,. That's the synchronous.

B

Request., but by introducing some of the latencies in this game,, you will change the way,. How.

B

Your system might operate in general., so I think that's that's the thing that people.

B

Always also forgetting., they think about the chaos engineering as ok,. So let's break things.

B

C

The breaking things is definitely fun,, at least not in production., but when you're doing.

C

That exploratory phase,, that is fun. and so, you know, it's I like to caution, people that.

C

That's not what it's about,, but at the same time again,. If, if what you're trying to do.

C

Is understand your systems,, it is fun and you should take time to break your systems.

C

To learn how they break and what that looks like,, particularly for folks that are maybe.

C

Not on the dev side, but more on the operations side, where you're on call and things will.

C

Break and you'll get a pagerduty alert and need to wake up in the middle of the night.

C

And fix it. having a better understanding about how things break and what that looks.

C

Like makes you a better engineer., so as much as you shouldn't just go breaking things,.

C

You should break things at least occasionally just to get more experience and better understanding.

C

There is a one gentleman, kyle kingsbury,, he known as the guy who breaks database for.

B

Databases for the money, profit and fun. so essentially his idea to kind of,, like, validate.

B

Some of the consistency claim of the data system., so databases and things like that.

B

And his blog is very popular., his tools are very popular. and it's actually very interesting.

B

Read and very interesting outcomes from the things that some of the software vendor can.

B

Promise you., you can learn more about the system and the way how he works,. He also like.

B

Provides suggestions for the authors of the systems, so they can improve this., but like.

B

His goal, not like break the system per say.. His goal is to confirm or debunk some of the.

B

Claims about the consistency,, which is a huge part in distributed systems world, in the.

B

Consistency of the data,, and especially with the applications and all this kind of stuff.

B

And I think the understanding,, like jason, mentioned, of how the system works, is important.

B

So let's talk about gremlin., so my understanding that name came from the famous movie from.

B

1984. joe dante's movie. and chris columbus wrote this the screenplay,, which was inspired.

B

By the tales of his grandparents, he was serving in the us, us air force. and sometimes when.

B

These planes are breaking, people tell this gremlins in doing things,. You know, the small.

B

Creatures that go in there and causing some mayhem., what do you do? There? like what does.

B

Gremlin do?: what is the thing that you folks are doing at gremlin?.

C

Yeah. so the lore, actually, yeah, it traces back.. I believe roald dahl, who's, a famous.

C

Author, wrote the first book about gremlins, and we have a classic book back in the gremlin.

C

Office. we have one office in san jose., but that is it.. It's that gremlins were essentially.

C

These mythical creatures that would help not only destroy technology, but actually would.

C

Help build technology as well. usually,. The destruction was when they were upset with.

C

Humans. and so that's where the name came from., but the tool itself is kind of similar.

C

We go in there and we help you inject failure in a safe, way,, safe,, controllable, way.. So.

C

Generally,, the way that that works is you would install the gremlin agent on your servers.

C

Or so we support both vms., so you would install our agent as a binary on your machines. linux.

C

And windows are supported. or you can do it on kubernetes., so we have a daemonset which.

C

Will install the agent for those who know kubernetes, daemonset,, so you'll get one gremlin.

C

Agent per node. from there, you'd use our backplane or control plane.. You can use that either by using the.

C

Web ui,, which I feel like a lot of folks use.. But if you're starting to do that, automation.

C

That ci/cd,, we have an api., so you communicate with our control plane. That would then send.

C

Out messages securely to all of the agents to figure out what targets are available and.

C

Then what targets you want to attack and what attacks you want to run. and so from there,.

C

It will communicate out and those agents will run those attacks.

C

We do have 11 types of attacks,, so we have resource attacks and those are things like.

C

Consuming cpu, disk space,, disk, io or memory., we have state attacks,. So that's shutting.

C

Down servers or containers, killing processes or changing the time on a machine., so changing.

C

A time is one of my favorite ones.. It's often overlooked, but useful for testing things like.

C

Ssl certificate expiration or, as viktor mentioned,, you know,. If you have a data, stream, oftentimes,.

C

That information is time critical., and so what happens when you get things that are.

C

Too far in the past, or maybe errant data that looks like it's coming from the future? and.

C

Then our third types of attacks are network attacks.. So that's introducing latency or.

C

Killing the network, corrupting or dropping packets and blocking dns.

C

Yeah,, that's that sounds like a really exciting tool that provides, you know,. The very extensive.

B

Number of things that can go wrong. and what about what the user see actually,? What's the...

B

I apologize. a little bit of noise in my background.

B

Are those your kids?

B

yeah, yeah.? They are like gremlins also. yeah,. They like gremlins, also like to go in the.

B

Like into do the things in my in my basement,, essentially so.

B

Um, all right, so, jason, in terms of like we know that there some certain tools that.

B

Are available to simulate certain the problems,, but like what? at the end, as a user, you will.

B

See, like how like do you visualize it?, like what's the outcome? you know,, we, we would.

B

Like to have some understanding, ok,, we broke the thing., but what do you think we would.

B

Learn from it? know and how gremlin helps developers to build a more reliable thing?.

C

Yeah,, so we often recommend that people follow the scientific, process., think of chaos, experiments.

C

Like science, experiments. so you'll start with observing your system and asking yourself.

C

A question., and often that question is just does this work the way I think it does,, but.

C

Usually it's a specific thing for does this work in the reliability monitor or is it reliable.

C

In the way that I think it is? and then to form a hypothesis, right?, what do I think.

C

Will happen? and then from there, inject the failure, analyze it and so I'll go ahead and.

C

Share my screen and we can actually,, let's follow that process for ourselves and do a.

C

Little bit of an experiment.

C

So we'll start off with this,. This is the google microservices demo. I like to use it.

C

Because it is a really classic microservices kind of application., it's an online store.

C

It's available here on github that you can see, and they've written each service in a.

C

Different language,, which is again pretty common. when you're working in teams, everybody.

C

Gets to choose how to build their service in what language and what way they want to.

C

Deliver those services. again,. This is what cloud native is all about., I'm going to scroll.

C

Down, though, and show you this architecture diagram that they've got.. So if we're thinking.

C

Of what is,, what are our systems? Look? Like,? Yours may not look exactly like this,, but.

C

You should have something: similar. you've got traffic coming in from the internet.

C

In this case, there's a load generator there, that's simulating real users.. It comes into.

C

Some sort of front end or through some sort of load balancer. in this case, kubernetes.

C

Is being the load balancer for us,, but it hits that front end service. and then from.

C

There we distribute out all of these requests. or you may have kong in here as well. That's.

C

Handling all of your api routing and and handling, coordinating all of that. but inevitably,.

C

As we talked about before,, you might have latency or you might have dead connections.

C

Or one of these service dependencies may go down., and so we have to think about what does.

C

That look like for our systems?: how do we handle that?? So when we're looking at this,.

C

The front end of the store looks like a basic shop.. It's got products, I can click into the.

C

Products like any good store., it's going to try to sell me on more stuff., so I've got.

C

The vintage camera here,, but I've also got a bunch of other things that it says I might.

C

Like or an advertisement, that's telling me that this is now 20 percent off, so I can click.

C

Around, or I could just go and buy this. and like any good online store,. I can add that.

C

To my cart- and it's got some info here so I can place my order and we see that we're.

C

Checking out just fine., so I can come into my monitoring tool.. I can see that I've got three nodes running in my.

C

Kubernetes cluster., I can see my response times for all of my services., so it's it's.

C

Operating fairly well. we've got redis running as well.. So if we look back and think of this.

C

Diagram of what we're running again,, we want to start thinking about things within our.

C

Distributed services, like time outs or retries or what happens when things go down. and so.

C

As I'm looking at this, well,, one of those things is,, you know,, if someone comes in and.

C

They're hitting that front end, that should be fine,. They should be able to check out.

C

So it looks like we have a direct connection to that checkout service and to that cart,.

C

Service. and from that checkout service, we've got payment and shipping and things. so,. You.

C

Know, I mentioned it's a standard store.. It's got that ability to upsell us or recommend.

C

Other things., but that looks like maybe not such a critical service, right?. It comes in.

C

It shows up on those pages,, but if I don't get suggestions for new products,, that should.

C

Be ok if I know I just want that camera or if I know that I just want some other product,.

C

I should just be able to to get those.. So, let's test that out. we've observed the system.

C

Let's have a hypothesis if I take down that recommendation system,, I should be able to.

C

Navigate to this page., if this doesn't appear,, that's fine,, we can still see our camera lens.

C

Here,, we can still add it to our cart,. We can still check out., so that seems like a.

C

Reasonable hypothesis., I think the other hypothesis is. Maybe we should see something in our monitoring.

C

System. not sure exactly what that might be.. Maybe our service, metrics here,, we'll see.

C

It get slow things like that., so we've done the first two steps, we've observed our system,.

C

We've formed a hypothesis, so now let's actually give it a try.. So this is what gremlin looks.

C

Like. gremlin, when you first come, in, it'll have a bunch of recommendations for you, things.

C

That you might want to consider as you're thinking about your system.. It's got a ton.

C

Of features,, but I'm going to really just focus on running an attack right, now., so I'll.

C

Click the attacks, say that we're running a new attack and we've got service discovery.

C

Enabled. so it's already gone out and it's looked at some of those services that are.

C

Available., so this allows me to just say,: let's select this recommendation, service.

C

It's going to already target things,, but if I wanted to add additional targets,, I could.

C

Do that choose a different cluster or a namespace,, but I'll leave those as the default. and then.

C

I can choose my gremlin or choose my attack., and so, as mentioned there,, there are those.

C

Three different types of attacks, resources, states or network.

C

Let's come in here and do a network attack and say we bump up latency,, so I'll select.

C

Latency., we'll only run this attack for 60 seconds and we can start injecting some latency.

C

So what happens?? Let's say,, if we inject one second thousand milliseconds of latency into.

C

This, there are a number of other controls that I can implement here,, but we can leave.

C

The defaults for all of them., one important thing to note, though, is we'll see that hostnames,.

C

Gremlin and the port is excluded. and that's because gremlin defaults to safety.. So if.

C

At any point, our agent, that's sitting on our machines, loses contact with the gremlin control.

C

Plane,, it's going to immediately halt and roll back.. All that said, everything default.

C

Should work just fine., so I'm going to go ahead and click unleash gremlin.. That's going.

C

To queue up this attack., so it's moved into a pending state and that'll start communicating.

C

Out to those agents that are sitting in my cloud environment on my kubernetes cluster.

C

They're going to pick up those instructions and target that recommendation service. so.

C

Now that that's queued up,, we can start to click around on our store and see what actually.

C

Happens. so clicking over to the vintage typewriter that took a bit of time. yeah,. So it's definitely.

C

Slower. there's a bit of latency in there.. We can check in our monitoring tool,. It does.

C

Look like that, I could see in my service metrics that we're starting to see that spike on that.

C

Recommendation service., but the nice thing is: we've confirmed that it looks like I can. Click around to products.

C

There's a slight delay,, but I can still get to that checkout.. I can place the order. so.

C

Overall, this looks like it's a pretty good experiment, we've confirmed that we've injected.

C

That latency. we've confirmed that our web store operates relatively well.. It's slow,.

C

It's probably annoying for users,, but it still works., and so hopefully you have situations.

C

Like this, where everything goes as expected., if we dive back into gremlin,, we can see here.

C

That it is still running and it's now flipped to successful.. So it's completed that attack.

C

So a few other things, right, we've done those first three steps now of observing the system.

C

Coming up with a hypothesis and injecting that latency., so one final thing that we can.

C

Do is to make some notes and observations.. We can say that system worked as expected.

C

And then we can add a link to any sort of dashboards or other things that we want and.

C

We can save those there. and the final thing now that we've analyzed that data recorded.

C

It here, we can share that out., so you're all with us. but yeah,. If you, if you were.

C

Part of my engineering, organization, well,, then we would share that out by an email.

C

That was pretty good and was pretty interesting to to see how easy it is to do and like how.

B

Visual, you can get, like visual response from the system that doesn't behave. and I I believe.

B

The other types of attack will look like something somehow similar. like,, you know,, maybe the.

B

Killing the process, like what we will see, what we expect. are we expecting to see like.

B

A file 2 error because it goes through a load, balancer and load balancer will not be able.

B

To reach the service, because the service was killed and like how fast it will recover?.

B

I guess that is all this information that we need to.. It's very resonates with the things.

B

That I like to tell my customers and users establish, like where you are right now and.

B

After., that's based on hypotheses that you implement., you are moving to the same sort.

B

Of conclusion., so I like this approach and it's very much resonates with the my way of.

B

Thinking in the way of thinking that I, like my customers and our users to to think about.

B

This. it's when, when you're going somewhere and saying, hey,, our system is slow, slow,.

B

How with this, you will see slow how like,, how much slower it became in what kind of.

B

Things. and as you saw in this demo,, it's like all about combination of multiple tools.

B

You know, the ones who will ingest the failures in tool. Will do, you know, tracing or observability.

B

Or metrics collection., so it's. many things that you need to employ in order to make these.

B

Things happen. well,, that was impressive. With the just the simple five minutes, you will.

B

Able to show how you can introduce this kind of new type of testing results, testing for.

B

Your system. usually,, I like to ask something about what the users or our listeners can read.. I like.

B

The reading books or listening audiobooks., it can be any book.. It can be fiction, science.

B

Fiction, or maybe like a business literature,, whatever you like or would recommend people.

B

To read. oh, wow, yeah., so I mean, there's. Definitely if you're interested more in gremlin, gremlin.com/community.

C

Is a great place to hit.: that's got a ton of resources., so we've got a huge slack. We've.

C

Got., you can sign up for gremlin for free over there and a bunch of resources. and we've.

C

Got a certification program for chaos; engineering as well in terms of books, though., one of.

C

My most favorite reads recently is how we got to now.. So how we got to now is a book.

C

By steven johnson. it's about innovation. and he talks about the adjacent possible and.

C

How, when we have innovation,, it doesn't come from nowhere.? It's we build successively on.

C

The innovations and the discoveries from people that came before us. and, as I think about.

C

Reliability and failure, I've started thinking about how that also plays in.. When we have.

C

Failures,, it's usually a bunch of different events., and so your catastrophes are that.

C

One little thing that added on to other failures?, so highly recommend that read it's all about.

C

Innovation. but as you do, that, start thinking about your own systems,, how they operate,.

C

How they got to,, where they're at based on the work that you've done and other folks.

C

Have done, and similarly, how they can fail.

C

thanks so much, jason, and thank you so much for being part of kongcast. and with this.

B

My name is viktor gamov and, as always, have a nice day.

B

Thanks to everyone out there for watching,, I hope you enjoyed today's episode as much.

A

As we enjoyed recording it. make sure to hit that like button now.

B

And be sure to subscribe, so you don't miss a new episode.

A

Don't forget to drop us a comment if you have any questions for today's guests, or if there's.

A

A topic you'd like to see us cover in the future.

B

And we will see you on the next episode of kongcast.