►
Description
🗣️ In our next Kong Builders Livestream, Andrew Kew is joining us to tell us more about the topic of Chaos Engineering.
With the move to distributed cloud architectures, it's becoming more critical to ensure that our platforms are designed for failure and behave the way we expect in these failure events.
Enter chaos engineering! The practice of injecting controlled failures into your system and observing how it responds to the experiment. Join this talk, where we will discuss the history of chaos engineering, why running everyday chaos engineering experiments is essential, and how you can incorporate this into your current CI/CD pipelines.
We will then apply this practice in a demo of a simple failure scenario in Kuma. The failure event will be injected using a chaos engineer tool called ChaosMesh.
=== timecodes ===
00:00 - stream starts
03:10 - intro
03:53 - introducing Andrew Kew
08:04 - introducing Chaos Engineering topic
17:24 - tools for trade: Chaos Mesh [https://chaos-mesh.org/]
20:38 - rant: the importance of Observability as a discipline
24:20 - demo time
28:35 - K6 for generating load
36:58 - bring some chaos: PodChaos policy
40:08 - simple test: Failure of service doesn’t impact consumer traffic
45:45 - first attempt - the consumer is getting 5xx responses
51:27 - we can do better: introducing Retry policy with Kuma mesh
57:56 - observing the results: no 5xx, only retries
59:59 - outro
A
A
A
B
B
Learning
technology,
we're
showing
I,
also
learning
I'm,
also
learning
some
stuff
and
also
sometimes
I'm,
bringing
some
awesome
guests
to
learn
about
some
cool
things
as
I'm
doing
today,
and
today,
I'm
extremely
excited
to
bring
the
Andrew
Q,
my
friend
from
Great
Britain
or
from
England
I.
Don't
know
how
you,
how
you
prefer!
Is
it
the
Great,
Britain
or
England.
B
B
So,
first
time
I
met
Andrew.
It
was
in
London
in
person
in
person
we
were,
we
were
chatting
in
hangout
and
he's
also
very
known
contributor
to
things.
What
we
do
here
at
Kong.
You've
probably
seen
him
on
one
of
the
episodes
of
the
Comcast,
and
you
probably
saw
him
speaking
at
at
Kong
Summit
so
but
before
we
jump
right
in
and
we're
talking
about
this
as
while
we're
chatting
also
let
us
know
how
everything
is
is
looking.
B
Where
are
we
coming
from
so
I
saw
some
of
our
Lego
Fox
regular
folks
are
coming
here.
So
let's
see
how
our
comments
section
is
working.
Tony
TV
comes
from
minitoba
minitoba.
Is
it
somewhere
where
in
Canada?
Hopefully
it's
not
too
cold?
Okay,
so
our
comments
work
also
write
down.
If
you,
if
you
come
in,
we
should
go
with
stream
in
multiple
places.
We
should
go
in
LinkedIn.
We
should
go
with
YouTube
twitch,
the
in
multiple
places
so
write
down.
B
Where
are
you
coming
to
so
right
now,
I
see
at
least
seven
people
at
least
seven
people
live
and
I
am
collecting
some
of
the
metrics
in
terms
of
like
where
the
people
coming
from
Andrew
tell
us
a
little
bit
about
yourself
like
what
brought
you
here
and
I
already
introduced
you
with
some
of
the
things
that
you
contributed
to
at
clunk
in
the
community
and
yeah.
So
what
do
you?
What
do
you
do?
What
are
you
up
to
these
days?
C
Yes,
yes,
thanks
thanks
Victor,
so
effectively
my
day,
job
I
founded
and
run
a
software
consultancy.
Here
in
the
United
Kingdom
we
specialize
in
microservices
and
integration
and
effectively
Kong
is
one
of
our
Tech
Partners.
So
we've
take
partnered
with
Kong
and
I
have
been
set
in
The
Residency
program
within
Kong
to
basically
help
out
in
the
customer
experience
team,
so
I'm
effectively
a
technical
account
manager
or
field
engineer
in
the
Kong
CX
team.
C
You
know
going
out
into
clients
and
helping
them
deploy
Kong.
You
know
helping
them
improve
their
architectures,
reviewing
their
designs,
but
then
also
helping
them
with
the
full
ecosystem
of
where
they've
deployed
it.
You
know
sort
of
anywhere
in
the
cloud
or
even
on-prem
as
well,
and
that's
probably
where
the
the
docker
whale
comes
into
it
as
well.
C
So
a
lot
of
what
I
do
is
kubernetes
based,
but
it's
not
sodi,
it's
just
in
kubernetes,
so
I
did
a
lot
of
kind
of
VM
work
as
well,
and
that's
probably
where
I've
come
from
in
the
last
the
last
couple
of
years.
You
know
working
from
bare
metal
through
to
VM
in
the
cloud
and
then
onto
the
containerization
as
well.
B
So
we
announced
this
the
this.
This
topic
for
today
is
chaos,
engineering
and
as
many
things
in
it,
you
know
many
things
in
the
software
development
World.
It
has
a
little
bit
cryptic,
sometimes
notation.
We
know
that
there
there's
the
companies
like
a
web
scale.
Companies
like
Netflix,
they
like
to
break
things
on
purpose,
and
this
is
what
we're
going
to
be
talking
today.
This
is
what
I
want
to
learn
and
hopefully
we'll
share
some
ideas.
So
Andrew
did
presentation
at
The
Conch
Summit
this
year
at
San
Francisco.
B
B
You've,
probably
seen
this
some
of
the
videos
that
do
share
in
on
the
Kong
Summit,
so
you
can
go
and
watch
this
keynote,
but
beyond
the
beyond
the
topic,
it's
going
to
be
on
the
topic
of
today
we're
going
to
talk
about
a
castle
machine
and
not
just
a
talking
about
so
the
Kong
Builders
is
where
we
doing
things
right.
So
we're
gonna
see
how
we
can
break
things
on
purpose.
So,
let's
start
just
to
bringing
everyone
up
to
speed
and
welcome
everyone,
and
we
see
lior
from
Tel.
B
B
B
Not
British
at
all,
but
I
love
tea,
I
prefer
I,
prefer
tea,
sometimes
to
Coffee
I
love
the
the
air
gray
like
of
the
yes
leaved
one
and
things
like
that.
Good
choice,
good
choice,
yep,
so
the
Sunil
is
jumping
just
in
front
of
gun.
It's
just
okay!
So
let's,
let's,
let's
get
to
the
point
like
okay,
what
kind
of
tool
you're
gonna
introduce
Sunil
embrace
yourself
is
gonna,
be
a
good
good
stream.
B
Okay.
So
what
is
Negroni?
Where
is
it
Neve?
Where
is
it?
Where
are
you
coming
from?
What
what
is
that
is
it?
What's?
It
is
the
country
or
town?
Let
us
know
all
right
so
Andrew.
Why
breaking
things
on
purpose
like
we
don't
want
to
break
things
right.
We
want
to
have
a
very
solid,
solid
environment,
so
why
why
to
bring?
Why
to
break.
C
I
think
I
think
that's
that's
under
the
the
whole
premise
of
it,
you
know,
is
resilience
and
and
ensuring
that
your
your
platforms
are
resilient
and
they
they
can,
or
you
know
the
behavior
to
particular
failures
and
that
that's
kind
of
the
whole
thing
with
chaos,
engineering.
You
know
it's
injecting
failure
in
a
controlled
environment
and
then
observing
the
behavior
of
your
platform
and,
seeing
you
know
how
does
it
behave
and
is
it
how
you
expected?
C
Hopefully
it
is,
but
if
it
isn't,
you
then
have
the
opportunity
to
you
know
resolve
it.
Reconfigure,
maybe
change
your
architecture.
If
it's
something
that
you
can
do
that
way
or
if
it's
something
you
can't
change,
then
at
least
you
know
what
the
failure
will
be
and
how
you
can
resolve
it
when
it
happens
in
production,
you
know
and
I
think
that's
the
whole
idea
behind
breaking
it
on
purpose.
You're
breaking
I
think
basically
make
it
more
resilient.
You
know.
B
Yeah
I
think
it's
the
also
good
good
thing
to
point
out
that
when,
when
are
we
talking
about
certain
technology,
where
we're
talking
about
certain
things,
people
like
to
come
and
say:
hey
how
I
can
install
this
chaos,
engineering
or
how
I
can
install
this
zero
trust
security
and
it
is.
It
is
very
interesting
thing
with
the
modern
software
and
the
Sunil
just
like
in
the
in
the
chat.
B
When
I
came
from
a
distributed
database
worlds
and
distributed
systems
world.
We
have
this
like
a
cap
theorem,
even
though,
over
the
time
when
my
thinking
involved
I
realized
that
this
theorem
is
actually
like.
A
two
I
would
like
a
two
like
a
narrow
minded
to
explain
complexities
of
the
system.
That
happened,
because
everything
there
would
be
different
trade-offs.
B
That
will
be
we'll
do
like
a
a
short
circuit
to
the
to
the
failure
and
it
will
not
propagate
things
or
things
with,
like
rate,
limiting
that
implements
between
the
nodes
of
the
of
the
of
our
systems,
so
I'm
passing
the
microphone
to
Andrew,
so
he
can
talk
a
little
bit
about
the
his
pronunciation
and
what
kind
of
setup
we
will
have
here?
Don't
forget
this
is
the
interactive
session.
This
is
the
difference
between
like
we
do
it's
not
a
webinar.
A
C
So
you
know
ensuring
that
that's
all
configured
you
know
you've
got
multi
multi-az.
So
you
know
your
service
is
running
in
multiple
zones.
I.E,
multiple
data
centers
and
that's
that's.
You
know
at
a
high
level
if
we're
just
looking
at
the
architecture.
That's
what
it
should
hopefully
help
you
with
as
well.
You
know
ensuring
that
you
do
have
that
resilience
and
then
self-healing.
You
know
that
you
maybe
configured
your
platform,
so
it
can
Auto
scale
or
it
can.
You
know,
manage
the
failure
effectively.
B
B
C
And
and
one
of
the
tools
yeah
one
of
the
tools
that
I've
been
using-
there
are
a
number
of
them
out
there,
but
the
one
I've
been
using
is
called
chaos
mesh
and
it's
a
really
really
nice
tool,
I'm
running
it
on
kubernetes,
but
I
think
you
can
get
it
to
run
on
on
VM
as
well
or
bare
metal,
but
it's
a
really
nice
tool
that
allows
you
to
Define
your
experiments.
So
you
know
the
the
failure
that
you
want
to
inject
into
your
system.
C
So,
just
to
kind
of
briefly
have
a
look
at
a
new
experiment.
You
know
you
can
see
from
a
kubernetes
standpoint,
there's
a
whole
raft
of
different
experiments
that
you
can.
You
can
build
out
of
the
box,
you
know
and
you
can
choose
from
a
combination
of
these
as
well.
You
can
schedule
them.
You
can
chain
them.
You
know,
there's
a
lot
that
you
can
do
and
effectively.
You
know
it
helps
with
I
guess
deciding
what
it
is.
C
B
B
You
can
find
a
playlist
with
the
Chromecast
episodes
where
I
was
talking
to
one
of
the
one
of
the
developer:
Engineers
developer
relations
Engineers
from
from
Gremlin,
which
also
in
other
chaos,
engineering
tool
that
you
can
and
there's
also
pretty
cool
demo,
so
just
something
to
to
to
consider
and
to
watch
in
in
the
meantime.
If
you
also
also
want
to
learn
something
like
this.
C
You
know
they
they
effectively
were
the
company
that
coined
the
term
chaos,
chaos,
engineering
and
it
was
you
know
their
migration
from
on-prem
or
bare
metal
servers
to
you
know
more
kind
of
cloud
distributed
systems
where
they
then
built
this
tool.
Chaos,
monkey
and
I
think
it's
still
available,
there's
still
a
GitHub
page
with
for
chaos,
monkey
that
you
can
go
and
and
download
it
and
use
it.
I
I
didn't
actually
try
it,
but
I.
B
B
You
cannot
control
something.
So
you
you
try
to
gain
back
control
during
the
chaos
you
know
during
the
failures
and
I
think
if
you
cannot
measure
it,
so
you
will
not
be
able
to
gain
control.
So
that's
why
integrating
with
some
sort
of
like
observability
component
also
is
important
a
bit
of
the
overall
discipline
right-
and
this
is
again
I'm
I'm
now
I'm,
jumping
in
front
of
the
gun
and
Andrew
will
will
have
something
to
talk
about
this.
C
You
know,
move
or
or
have
the
knowledge
of
when
it
does
happen.
You
know
how
to
resolve
it.
You
know
I
mean
I,
think
having
that
knowledge
is
especially
for
your
support
team.
You
know
the
teams
that
are
probably
going
to
get
woken
up
at
two
o'clock
in
the
morning
when
there's
an
outage,
you
know
as
long
as
they've
got
playbooks,
you
know
a
Playbook
and
how
to
resolve
this
particular
issue.
C
Then,
actually
it's
not
the
end
of
the
world
or
you
know
not
a
massive
impact
if
they
can
get
things
back
up
and
running
it's
when
you
don't
know
what
happened
and
then
you've
got
to
start
investigating
it.
You
know
and
effectively
like
if
I
go
back
to
the
issue
that
Netflix
had
originally
I
think
was
2008
when
they
had
their
major
outage.
It
took
him
three
days
to
work
out
what
the
problem
was,
and
they
probably
didn't
have
a
really
good,
observability
layer.
You
know
to
be
able
to
see
you
know.
C
B
B
You
have
infinite
bandwidth,
so
you
will
be
kept
by
your
cloud
provider
most
likely
or
you
will
be
capped
by
your,
but
your
network
team,
so
network
is
secure
and
the
network
is
most
importantly
homogeneous
and
when
we
will
be
running
into
multiple
environments,
so
like
different
regions
or
maybe
things
around
some
of
the
bits
are
running
inside
kubernetes.
Some
of
the
bits
are
running
inside
bare
metal
and
those
systems
need
to
communicate
somehow
so.
C
B
C
C
A
really
really
nice
feature,
you
know,
being
able
to
load
balance
between
clusters,
not
just
load
balancing
between
pods
in
the
same
cluster.
So
a
really
really
nice
feature
that
I'll
be
hinging
off
of
then
observability,
which
is
you
know,
one
of
the
things
we
spoke
about
as
well,
and
I've
got
Prometheus
running
in
each
of
the
Zone
clusters
and
then
I've
got
a
Prometheus
in
the
global
cluster
up
at
the
top
and
using
that
to
effect
effectively
push
the
the
metrics
up.
C
C
Yes
yeah,
so
k6
is
a
really
nice
load
testing
tool
and
you
can
use
it
to
effectively
generate
load
and
then
and-
and
there
are
a
number
of
ways
you
can
do
it
so
it
it
works
by
writing
very
simple
JavaScript
files,
and
it
has
a
whole
lot
of
libraries
with
it.
Where
you
can
do
things
like
send
you
know
consistent
or
constant
load,
you
can
do
multiple
virtual
users.
C
C
C
And
that's
effectively
the
the
JavaScript
very
simple
as
well.
You
know
I'm
doing
an
import
for
a
particular
Library
I'm
setting
some
options
and
then
I'm,
basically
just
calling
my
my
get
on
an
endpoint
and
it's
as
simple
as
that.
But
if
you
have
a
look
at
the
scenario
that
I'm
I'm
effectively
running
I,
think
that's
for
this
use
case.
C
C
I
wasn't
trying
to
test
out
the
performance
in
this
case,
but
you
know
it
at
least
gave
me
something
to
be
able
to
generate
some
some
load
on
the
cluster.
You
you,
if
you
want
to
do
something
like
you
know,
test
out,
Auto
scaling
for
argument's
sake,
or
you
know
how
many
requests
or
how
many
users
can
concurrent
users
can
hit
my
service
in
one
time.
You
know
you
can
tweak
these
numbers
engage
in
a
much
higher
TPS,
much
higher
and
virtual
users
as
well
and
actually
start
observing.
B
Well,
I
I
I
get
to
use
this
tool
called
Locust,
but
it
uses
python.
So
I
think
the
JavaScript
is
more
for
me
would
be
more
interesting
language
to
use.
Not
a
huge
fan
of
python,
but
I
would
like
to
remain.
I
would
like
to
remind
that
there's
still
a
video
that
needs
to
collect
100
likes,
so
I
will
start
doing
some
of
the
performance
testing
live
sessions
where
we're
going
to
be
bringing
Kong
and
or
other
components
down
through
the
using
those
tools.
B
Speaking
about
this
data,
so
do
you
do
you
running
this
somewhere
on
some
of
the
environments?
So
you
can
point
this
to
any
any
location
or
there
is
a
I,
don't
know,
maybe
there's
a
managed
service
that
allows
to
just
like
upload
your
scenario
and
they
will
be
able
to
handle
all
this
infrastructure
growth
for
you.
B
Yeah
k6
like
if
I
want
to
do
distributed,
distributed
performance
performance
measurement,
so
I
I
don't
want
to
do
like
distributed
denial
of
service
type
of
attack,
but
you
know
sometimes,
when
I
run
my
systems
in
open
internet
I
want
to
have
open
internet
system.
That
also
will
try
to
you
know,
generate
some
load.
C
Yeah,
absolutely
I,
don't
I,
don't
know
of
k6
that
that
they
do
do
that.
You
know
this
is
running
locally
on
my
machine
and
just
hit
hitting
a
an
end
point.
You
can
have
it
running
in
your
cluster,
so
I
guess
effectively.
You
know
you
could
have
a
container
or
pod
running.
You
could
have
multiple
pods
running
and
getting
all
of
them
to
maybe
pull
things.
So
you
could
probably
create
yourself
a
distributed
load.
Testing
system
within
you,
know,
eks,
or
something
like
that.
C
K6
themselves
have
actually
been
brought
into
the
grafana
labs,
so
I'm
sure
there'll
be
more.
You
know
we'll
see
more
from
them
in
in
in
the
future,
and
you
never
know
there
may
end
up
being
something
you
know:
grafana
has
their
Cloud
offering
their
SAS
offering
they
may
end
up
bringing
something
in
with
k6
and
giving
you
some
kind
of
you
know
SAS
offering
for
performance
testing
yeah.
You.
B
B
C
So
you
know
you
can
have
a
look
at
your
performance
tests
and
as
they're
running
you
know,
you
can
get
sort
of
live
information
as
to
what's
going
on
and
and
what
tests
that
you
know
the
data
around
those
tests
as
well,
which
is
really
nice
if
you're
getting
any
errors.
So
you
know
again
centralizing
it's
a
central
observability
of
on
your
platform,
but
also
your
load
testing
as
well
and
then
being
able
to
I
guess
marry
all
of
this
up
with,
because
you
know
you
can
build
whatever
dashboards
you
want
in
grafana.
C
C
C
C
You
know
this
is
where
we'll
start,
having
a
look
at
a
certain
aspects
to
I
guess
what
I
want
to
do
to
show
today
and-
and
you
know
it
gives
us
again
visibility
as
to
what
things
look
like
you
know,
you
can
see.
I've
got
some
failures
here
and
we
can
kind
of
talk
through
that
and
maybe
even
run
it
live
as
well.
So
you
can
see
it
and
and
see
what
it
looks
like
and
and
and
and
so.
B
C
Exactly
so,
the
I
mean
these.
These
panels
here
are
actually
coming
from
the
out
of
the
box.
Observability
you
get
with
Kuma
mesh,
which
is
great,
and
then
some
of
them
were
coming
from
k6
as
well,
so
I
kind
of
yeah,
I
kind
of
mixed
and
matched
like
I
was
saying
between
some
of
the
k6
ones,
some
of
the
kuma
ones,
and
then
you
know
from
that.
It
gave
me
a
good
idea
as
to
what
was
going
on
and
what
the
problem
was.
C
The
other
thing
I
actually
wanted
to
talk
about
quickly,
so
just
having
a
look
at
a
chaos
mesh
and
the
one
thing
I
really
like
about
it
is
chaos.
Mesh
can
actually,
your
experiments
can
be
defined
in
yaml,
which
is
really
really
nice.
So
you
can
take
those
experiments
and
you
can
apply
them,
so
you
know
in
a
git
Ops
fashion,
and
then
you
can
re-run
these
via
your
crcd
pipelines.
C
So
you
know,
rather
than
having
tools
where
you've
maybe
got
to
run
a
CLI
tool
or
you've
got
to
have
something
else
added.
You
know
once
you've
got
this
installed
into
your
cluster
effectively
just
apply
in
this
yaml
and
we'll
do
that
as
well.
You'll
see
it
running
the
experiments,
and
in
this
case
this
is
a
pod
chaos.
So
what
this
will
effectively
be
doing
is
the
action
of
a
pod
failure,
so
it
pulls
a
pod
down
and
it
runs
for
a
duration
of
three
minutes.
B
A
good
question
on
this
one,
so
this
chaos
mesh
we're
going
to
be
it's
going
to
be
controller
that
installed
inside
the
kubernetes
cluster.
That
will
be
handling
some
of
those
like
a
kubernetes,
specific
issues
right
and
when
we
have
this
policy,
this
policy
will
be
applied
and
this
controller
will
be
ultimately
controlling
how
those
pods
will
go
up
and
down
things
like
that.
Right.
A
C
Yeah,
that's
exactly
right!
That's
exactly
right!
Then
I
can
probably
show
you
that
as
well,
if
I
jump
into
K9
as
well,
anyone
out
there
who
is
not
using
canine
I
highly
recommend
it
as
a
as
a
tool
for
viewing
your
clusters,
so
not
a
complete
replacement
for
a
cube
cuddle,
but
it
definitely
speeds
up
my
day-to-day
music
of
clusters.
So.
C
Definitely
I
I
think
I
got
the
tip
from
you
seeing
you
use
it
at
one
of
our
talks
that
we
did
and
then
yeah
I've
haven't
looked
back
since
so
yeah
great
tip
thanks
so
yeah
just
having
a
look
here.
These
are
the
pods
that
are
deployed
when
you
deploy
chaos
mesh
and
effectively.
You've
got
a
dashboard.
C
You've
got
some
demons
as
well
that
are
effectively
running
on
each
of
your
notes,
so
they're
running
as
demon
sets
and
then
you've
you've
just
got
a
full
redundancy.
Some
controller
managers
as
well.
You
know
those
are
the
ones
that
are
effectively
listening
out
for
any
injection
of
experiments
and
then
effectively
the
demons
will
go
out
and
do
whatever
it
is.
You
need
to
if
there's
anything
at
the
point
of
needing
to
do
something
at
a
server
level.
C
You
actually
have
to
install
something
on
the
the
physical
node.
So
if
it's
kubernetes
your
worker
node,
you
you'll
need
to
build
a
custom
Ami
for
that
worker
node
with
that
additional
Library
added,
so
that
it
can
do
anything
if
it's
if
it's
any
kind
of
experiment
or
failure
almost
below
the
cluster
level.
So
there
is
that
one
consideration
that
you
would
need
to
take
into
account
as
well.
B
C
Yeah,
that's
right,
that's
right!
So
having
a
look
at
just
a
simple
test
as
well.
You
know
so
in
in
this
case,
following
those
principles
that
I
was
talking
about
before
you
know,
defining
a
hypothesis.
So
basically,
a
failure
of
a
service
within
a
Zone
mesh
should
have
no
impact
to
Consumer.
So
no
no
errors.
So
from
a
consumer
standpoint,
they
don't
what
really
want
to
know
about
any
failures
within
your
mesh.
C
The
experiments
which
I
was
I'll
show
you
that
yaml
within
chaos
mesh
is
injecting
pod
failure
into
the
zone
and
then
effectively
we'll,
observe
it
and
then
work
out
what
it
is.
We
can
do
to
improve
it,
rerun
the
experiment
and
then
hopefully
get
a
better
output
at
the
end
of
that
that's
effectively
what
we
we
will
do
so
just
showing
you
here.
C
Yeah,
that's
right.
If
I
go
back
to
here
this
cluster
on
the
left
is
my
zone,
one
cluster
and
then
on.
The
right
is
the
zone
two
cluster
and
then
the
global
one
was
the
one
up
at
the
top,
so
we'll
be
basically
calling
everything
into
this
one
zone
and
it
will
be
load
balancing
between
the
two
zones.
C
C
C
C
You
need
to
push
it
into
a
influx
DB
instance,
which
I've
just
got
running
in
that
same
one
instance
as
well,
so
I'm
just
going
to
come
in
here
and
if
we
go
to
load
testing
last
five
minutes
and
yeah
and
as
you
can
see,
we're
starting
to
get
some
data
coming
through
as
well.
So
we
can
see
that
our
experiment
is
effectively
running
or
our
load
testing
is
effectively.
A
B
C
C
It's
probably
not
going
to
be
as
as
a
nice
diagram
as
the
one
I
previously
showed
you
just
because
I
needed
to
start
things
at
the
beginning.
So
you
can
actually
see
here.
There's
some
errors
coming
through
there
and
there
might
even
be,
by
the
time
I
took
to
show
you
everything
that
this
guy's
almost
finished
as
well.
So.
B
So
right
now
this
is
the
this
is
the
data
that
we
captured
from
where
so
from
the
k
k6
the
the
low
generated,
because
initially
you
said
initially,
you
said
that
our
consumers
of
API
should
not
feel
if
one
of
the
nodes
went
down.
So
how
how
you,
let's,
let's
explain
this
the
numbers
a
little
bit,
so
we
still
have
some
500
type
of
errors.
C
Yeah,
exactly
I
think
that's
kind
of
the
point
of
what
I'm
trying
to
show
here
so,
like
I'm
running
a
test
where
we
are
getting
some
errors
and
effectively
the
the
consumer
is
seeing
the
errors,
because
there's
an
issue
with
the
the
cluster
and
that's
that's
kind
of
the
premise
of
what
I
guess
and
what
I'm
showing
here
is.
What
we
will
do
is
then
reconfigure
humor
with
a
policy.
It
has
something
called
a
retry
policy
and
you
can
affect.
C
B
While
you're
doing
this,
I
will
explain
one
more
time
what
happens
here.
So
the
idea
of
this
experiment
is
that
we
gonna
be
breaking
this
system
and
see
what
kind
of
behavior
we're
going
to
be
observing,
because
in
future
we
want
to
improve
this
Behavior,
because,
right
now,
if
consumer
hits,
the
Pod
that
is
restarting
or
Port
is
unavailable
for
30
minutes.
Gateway
will
because
Gateway
fronted
this
service
Gateway
will
throw
500
there.
We
want
to
avoid
this,
and
this
is
going
to
be
our
next
step.
B
C
Yeah,
exactly
and-
and
you
can
see
here,
the
failures
are
starting
to
come
through
effectively.
You
know-
and
this
was
this
was
me
talking
about-
observing
the
behavior
and
then
seeing
whether
you
can
improve
what
you've
got,
and
in
this
case
you
know
very
simple
example.
In
this
case,
the
improving
is
going
to
be
applying
a
policy
to
my
mesh
that
effectively
handles
the
failure
better.
You
know
in
this
case
it's
something:
that's
you
know
internal
to
to
to
humor
to
Kuma
mesh,
but
it
could
be.
It
could
be
anything
else.
C
It
could
be
any.
You
know
this
was
just
the
the
scenario
or
the
hypothesis
that
I
chose
yeah
yeah.
You
can
start
seeing
those
500
errors
are
increasing
I'm,
starting
to
get
failures
coming
through,
and
this
these
failures
now
that
they're
greater
than
zero
effectively
means
the
consumer
is
getting
errors,
because
every
every
now
and
again
it'll
be
trying
to
load
balance
to
a
pod
that
doesn't
exist
or
a
pod.
That's
been
pulled
down
and
a
effectively
returning
that
500
error
back
to
the
consumer,
and
that's
what
you
don't
want.
C
You
know
exactly
because
exactly
you
don't
yeah
you
at
the
end
of
the
day,
the
the
fact
that
you
are
load
balancing
between
two
clusters
and
there's
an
issue
between
your
clusters.
You
know
whether
it's
the
service
gone
down
or
the
comms
between
your
two
Zone
clusters.
You
know,
that's
that's
I!
Guess
implementation
detail
that
that
you
as
a
platform,
support
or
or
company,
should
be
managing
yourselves.
You.
B
Know-
and
another
thing
is
that
since
Andrew
mentioned
about
a
little
bit
about
implementation
details,
usually
if
we're
talking
about
this
as
a
this
experiment
needs
to
be
by
the
way
this
experiment
needs
to
be
constant.
This
needs
to
be
part
of
some
sort
of
integration,
testing
phase
of
your
application
when
you're
rolling
things
out,
because
this
is
still
place
where
it
can
affect
certain
things.
B
If
they're
not
using
any
like
software
defined
Network
systems
like
service
mesh,
they
would
have
to
go
and
Implement
those
kind
of
like
logic
in
terms
of
how
to
do
proper,
pushback
or
maybe
Implement
a
built-in
like
circuit
breakers,
some
of
the
tools,
some
of
the
Frameworks
some
of
the
languages.
They
already
include
the
implementation
of
those
tools,
but
with
service
mesh.
That's
what
we
will
I
I
hope.
C
B
Have
ability
to
implement
those
as
a
part
of
some
sort
of
like
a
pipeline,
some
sort
of
cicd
pipeline
where
continuous
integration
pipeline
that
will
be?
You
have
a
run,
this
hypothesis
and
you
collect
the
data
in
your
integration,
testing
phase
or
your
development
phase,
and
after
that,
for
your
infrastructure.
When
you're
rolling
this
out,
you
will
be
implementing
those
policies
that
would
allow
to
mitigate
mitigate
this
type
of
failures.
C
Yeah
absolutely-
and
you
know
the
way
I
see
it
is
I,
see
it
as
units
testing
for
your
platform.
You
know,
as
as
part
of
your
your
cicd
pipelines
you,
you
run
your
integration
tests.
You
run
your
unit
tests
for
all
of
your
code
for
your
micro
services,
but
you
know
who's
actually
running
those
kinds
of
unit
tests
for
your
platform
to
ensure
that
you
know,
maybe,
if
someone
changes,
especially
if
things
are
you
know,
built
with
infrastructures
code-
is
adding
a
new
component
to
its
breaking
something
or
have
we
actually
configured
it
correctly.
C
You
know
with
within
the
declarative
configs
of
your
infrastructure
as
well,
and
that's
that's,
how
I
see
it
being
used
and
actually
you
know
running
it
on
a
daily
basis
or
even
on
a
change
by
change
basis.
If
you're
able
to
depends
on
how
large
it
is
the
the
platform
that
you
need
a
stand-up
takes,
you
know
it
could
be
an
overnight
sort
of
test
that
that
runs
and
sort
of
runs
your
whole
Suite
of
tests,
and
then
you
get
the
results
in
the
morning.
You
know.
B
C
What
it
will
do
is
if
there
is
a
service,
that's
out
so
in
other
words
the
pod,
that's
not
there,
it
will
try
a
couple
of
times
and
then
it
will
revert
the
traffic
to
the
service
that
does
exist
so
effectively
within
the
mesh.
You
are
then
handling
that
error.
So,
like
you
were
saying
the
consumer,
then
shouldn't
be
able
to
see
any
issues.
You
know
and
that's
that's
effectively
what
we
will
be
doing
now
and
then
I'll
run
that
and
just
and
just
kind
of
show
you
what
the
results
look
like
as
well
and.
B
B
B
Sometimes
they
don't
need
to.
Even
you
know,
talk
about
this
deal,
which
is
that,
like
the
latest
greatest
data
or
latest
greatest
update
response
will
be
blocked.
So
again,
it's
it's
the
to
the
point
where
you
need
to
agree
with
your.
You
know
across
the
engineering
across
the
product
leaders
like
what
is
the
user
experience
you're
trying
to
get
to
your
users
and
what
would
be
suitable
and
appropriate
response
and
the
user
experience.
B
Sometimes
you
even
can
put
like
if,
if
you're
doing
this
API
as
a
product,
those
latencies
you
can
even
include
on
your
SLA,
saying
like
sometimes
like
the
90
90
percentile,
you
will
get
X
response
permit,
meaning
that
you
still
have
a
Headroom
for
for
certain
errors
like
when
the
latency
increases,
so
those
things
need
to
be
agreed
between
people.
That's
the
people,
part
technology
wise.
You
got
you
covered,
but
the
experience
wise
and
how
those
numbers
would
be
interpreted.
It
needs
to
be
handled
across
the
teams.
C
Yeah,
absolutely
absolutely
so
effectively
what
I've
done
is
I've
applied
a
retry
policy
and
I'm
just
going
to
jump
on
to
Cluster
under
c
and
I've,
effectively
got
a
retry
policy
now
running.
This
is
on
my
zone,
one
cluster,
so
you
know
with
a
multi-zone
configuration
with
Kuma,
you
apply
it
at
the
global
control
plane
and
that
then
pushes
it
down
to
both
of
the
Zone
Zone
clusters
as
well.
B
C
There
we
go
so
if
I
do
full
screen
yep.
So
that's
effectively
it.
You
know
it's
a
it's
a
particular
crd.
That's
that's
Akuma
built
one
for
for
retries
and
I've.
Basically,
just
just
told
it
to
there's
a
lot
of
default
settings
here,
but
I've
effectively,
just
told
it
to
retry
five
times
on
HTTP
and
and
that's
effectively
what
it's
going
to
do,
because
it's
not
actually
using
any
of
the
other
protocols
so
effectively
it's
going
to
try
five
times.
B
Recall
interesting
thing
here
that
one
of
the
things
that
I
was
also
also
was
talking
about
the
cascading
failures.
Retries
are
good
unless
those
retries
are
also
controlled.
As
you
can
see
here
in
there
is
a
back-off
policy
as
well
available,
so
you
don't
want
to
just
do
boom
boom
boom.
If
the
retry
failed,
it
will
try
to
wait
certain
moment
of
time
and
after
that,
try
it
one
more
time.
So
it's
it's
better
to
have
this
kind
of
I'm
a
better
math.
C
C
C
C
You
know
it's
being
resolved
by
humor,
but
what
we
will
see
is
the
retry
count
and
that
you
will
see
it
trying
to
retry
a
particular
service
so
where,
before
which
I
can
zoom
into
after
this,
there
were
no
retries
because
their
policy
didn't
exist.
So
you
know
you
can
get
some
stats
on
what
it
is
doing
internally
in
the
cluster
as
well,
which
is
which
is
really
nice,
so
yeah.
Let
that
go
for
a
second
and
then
what
I'm
going
to
do
is
then
apply
the
sky.
C
And
if
we
jump
back
to
here,
you
should
see
yeah,
there's
my
with
retry
policycon
builders.
So
that's
again
same
experiment,
pod,
pod,
failure
running
for
three
minutes
and
then
yeah
jumping
into
here.
We
should
hopefully
start
seeing
no
failures
and
and
and
some
retry
counts
coming
through
and
then
that'll
effectively
show
you.
C
C
The
behavior
here
we
go,
there's
some
retries
coming
through
observed,
Behavior
we've
gone,
this
is
what's
happened,
that's
not
what
I
want
and
I've
improved
what
I've
got
and
then
re-run
it
again
and
now
we
can
see
you
know
well.
These
retrials
are
coming
in.
You
can
see
that
there
there
are
only
200s
coming
through
there's
no
500,
there's
no
errors,
no
failures
coming
through
and
effectively,
you
know
improved
I,
guess
the
response
of
of
our
platform.
C
B
I
think
it's
great
I
think
it's
awesome
and
I
think
this
is
actually
you
know
the
covers
many
aspects
of
of
today's
conversation
and
I
would
like
to
say
that
Andrew,
if
you
have
any
like
a
pointers
where
the
people
can
pick
up
your
demo,
if
it's
like
available
somewhere
on
GitHub,
if
they
want
to
play
around
or
you
know
how
to
you
know,
learn
more
from
from
from
this
from
the
session.
Maybe
you
have
published,
slides
somewhere
so
that
people
can
like
take
a
look
on
on
those
things.
C
Yeah
I
I
think
if
you
want
to
do
that,
reach
out
to
me
on
LinkedIn
I
haven't
yet
published
the
this
demo
on
on
GitHub,
but
you
know
that
that'll
be
something
I'll
be
doing
in
the
next
couple
of
weeks.
So
I'll
be
posting
that
on
on
GitHub,
but
yeah
reach
out
to
me
on
LinkedIn,
more
than
happy
to
you
know,
show
you
this
point
you
in
the
right
direction
and-
and
just
you
know,
get
you
started
ready
in
in
this
domain.
B
Yeah,
absolutely
so,
and
always
don't
forget
to
subscribe
to
our
YouTube
channel,
and
this
is
the
Kong
Inc,
where
you
can
find
every
material
every
possible
things
with
regards
to
your
Cloud
connectivity,
apis,
API,
connectives,
kubernetes
and
all
these
kind
of
things,
and
again
thank
you
for
those
of
you
who
were
with
us
in
this
session
live
in
asking
the
question
and
those
of
you
who
watching
this
offline.
You
can
always
write
down
the
comments
and
ask
any
appropriate
questions
subscribe
to
this.
B
We
also
working
on
enabling
slack,
so
you
can
go
and
come
in
the
chat
with
us,
so
stay
tuned
for
those
announcements
and
yeah
so
for
for
today,
again
I'm
super
excited
to
bring
you
some
of
the
guests
gives
give
some
of
the
give
us
some
reactions.
If
you
like
the
show-
and
hopefully
Andrew
got
his
the
the
payment
increased
and
he
will
get
the
the
some
Karma
points
for
being
awesome.
B
Kong
Champion
again
Kong
Champion
program
is
this:
is
the
our
community
program
inside
the
Quan
we're
recognizing
some
individuals
who
help
others
to
to
learn
about
konk,
about
Cloud
connectivity
about
apis
and
like
people
like
Andrew,
like
speaking
at
the
meetups,
and
sharing
some
of
the
knowledge
and
coming
up
with
some
some
cool
ideas?
It's
a
great
Community
for
individuals.
If
you
want
to
join
us,
please
reach
out
and
show
some
of
some
of
your
cool
stuff
around
the
Conch,
and
we
will
nominate
you
for
the
next
round,
so
Andrew.