►
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Running Resilient Workloads with Istio - Matt Turner, Tetrate
Join us for KubeCon + CloudNativeCon in San Diego November 18 - 21. Learn more at https://bit.ly/2XTN3ho. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
Remember how cool Kubernetes seemed when you first started using it? A simple, easy API for scalable compute in any cloud: just a Deployment and a Service and you’re done! But as you use it more, you learn that this isn't really enough. A production system needs requests and limits, liveness checks, HPAs, PDBs, PSPs, etc. The same is true for Istio, which can solve a lot of the problems with microservices out of the box, but isn't magic. When you get beyond playing with bookinfo, more configuration is needed to get the most out of it. In this talk I’ll show you how to: - Identify app versions, deploy canaries and run A/B tests - Set timeouts - Configure retries, with exponential backoff - Enforce rate limits - Enable circuit breakers - Inject faults for testing I’ll also cover a couple of the big security features: - Enabling mTLS - Using service-to-service access control lists (RBAC)
https://sched.co/Nrps
A
Thanks
for
coming
to
my
talk,
I
am
Turner
from
London
in
the
UK
I
guess
I've.
You
are
using
the
translation,
so
I'll
try
to
go
a
little
slow,
I'm
going
to
be
talking
about
sto
I'm,
going
to
be
talking
about
running
resilient
workloads
using
sto.
So
this
is
looking
at
a
higher
level
at
sort
of
applications
and
how
you
can
use
SEO
to
solve
some
of
the
problems
you
get
with
distributed.
Applications
distributed
micro
services.
A
A
Ok,
most
people
that
coop
Khan,
who
is
who
is
using
Sto
in
their
production
system,
ok-
and
everybody
else-
has-
is
look
to
this-
do
interested
in
Sto,
ok,
cool!
Well,
hopefully
this
is
this
is
some
useful
information
for
you,
so
just
a
quick
I
can't
see
the
slides
from
there
a
quick
introduction
about.
You
know
why?
Why
we're
looking
at
this
think
about
what
your
business?
What's
from
the
software
you
write
to
think
about
the
value
you're
delivering
to
users?
A
What
is
the
value
to
the
users
and
therefore
the
value
to
the
business
of
what
you're
building?
Well,
yes,
you
want
your
software
to
be
good.
You
want
it
to
be
high
quality.
You
want
it
to
to
sell
for
a
lot
of
money.
That
is
your
top
line
right.
You
want
lots
of
income,
you
want
it
to
run
as
cheaply
as
possible.
That's
the
bottom
line,
so
you
want
to
keep
your
expenses
down.
A
Kubernetes
helps
you
with
your
bin
packing
maximizing
utilization
of
your
cloud
resources,
but
also
what
is
important
to
a
business
is
the
time
to
market.
So
add
the
the
speed
of
experimentation
and
it's
raishin.
So
how
fast
you
can
develop
a
new
feature,
get
it
out
to
a
user.
You
know
test
it
find
out
whether
it's
what
the
user
wants
and
then
use
that
to
influence
your
next
sprint
or
whatever
right
and
then
get
useful
feedback
from
your
actual
users
on
your
product,
and
this
reduces
the
risk
of
building
software.
A
This
reduces
the
risk
of
writing
code
for
two
years
in
isolation
and
then
releasing
it
and
finding
out
that
it's
not
what
anybody
wanted
and
you
can't
set
it.
So
this
is
really
important
and
there's
been
a
lot
of
work
in
software
in
the
last
20
years.
Maybe
to
get
to
that
point,
so
we
had
scrum,
which
says:
don't
play
more
than
two
weeks
in
advance,
because
you
don't
have
the
evidence
and
we
also
changed
software
engineering.
So
we
took
the
monolithic
applications
that
we
had
and
we
we
broke
them
apart
right.
A
So
this
is
a
rock
that
is
cracked
for
anybody
who
doesn't
speak
English
or
like
Latin
monolith.
That
means
single
rock
big
rock,
so
we
took
them.
This
is
why
the
picture
it's
a
bad
joke.
We
took
the
big
rock
and
we
made
it
into
lots
of
little
rocks,
so
we
have
microservices,
and
that
means
that
we
can
develop
those
in
parallel.
The
code
bases
are
smaller,
they
are
easier
to
edit
and
we
can
release
the
software
separately
right.
A
So
if
I
want
to
make
change
to
one
part
of
my
application,
ie
make
a
change
to
that.
Microservice
and
I
can
release
it
and
I
don't
have
to
wait
for
everybody
else.
So
I
get
my
feedback
really
fast
and
that's
that's
how
we
why
we
decided
to
write
microservices
one
of
the
reasons,
but
there
is
a
joke.
You
know
now
that
we've
broken
the
monolith.
How
do
we
know
that
we
haven't?
Don't
just
have
a
distributed
one
of
this,
so
the
joke
says
in
microservices.
No,
no.
What
we've
got
is
a
completely
different
pattern.
A
We
have
a
distributed
monolith,
so
I've
taken
all
the
code
that
I
had
before
you
know:
I've
separated
my
concerns,
I'm,
a
good
software
engineer.
I've
got
dependency,
injection
and
I've
separated
my
code
into
different
Java
namespaces,
behinds,
nice
strong
interfaces
and
they
called
each
other
right.
We
knew
how
to
write
software
in
the
in
da.
It
wasn't.
Nothing
is
new,
but
now
with
microservices
I
split
that
I
put
them
all
in
a
different
pod
and
I
run
them
on
kubernetes.
So
we
have
the
same
code
and
now
it's
spread
out.
A
The
problem
is
what
used
to
be
a
function,
call
between
namespace,
a
and
namespace
see
that
could
never
fail.
It's
now
a
call
over
a
network,
so
there's
a
whole
load
of
ways.
It
could
go
wrong.
The
network
could
be
on
fire.
The
computer
that
pod
C
is
running
on
could
be
on
fire.
The
data
center
could
be
on
fire,
so
we
have
all
of
the
same
problems
as
before,
all
of
the
bugs
in
this
code,
and
there
are
always
bugs
and
a
whole
load
of
new
failure
modes
which
we
didn't
have
before.
A
So,
if
my
micro
services
look
like
this,
you
know
one
front
end
cause
some
backends
that
all
depend
on
each
other.
Then
if
one
goes
wrong,
I
can
start
to
get
these
cascading
failures.
If
I'm,
not
careful,
this
is
very
complicated.
This
is
an
emergent
property
of
this
system
right,
so
one
one
service
fails.
Nothing
else
can
reach
it.
Some
of
those
don't
handle
that
error
and
they
fail
some
of
them
block.
A
While
they're
you
know
waiting
and
they
sit
there
forever
and
I
get
all
these
kind
of
cascading
failure
modes
that
I
didn't
have
before.
So
this
is
a
distributed
monitor
if
this
is
a
monolith
where
I've
just
thrown
it
into
different
pods
and
scaled
it
up,
but
not
really
thought
about,
and
what
that
means
for
the
quality
of
my
system.
So
we
have
a
way
of
solving
this
and
you
know
most
of
you
know
what
sto
is
so
I
won't
spend
long
on
that.
A
Well,
one
of
the
ways
we
can
solve
this
is
with
a
service
mesh,
so
the
service
mesh
comes
along
and
puts
a
proxy
I
will
be
demonstrating.
Sto
and
sto
uses
the
Envoy
proxy.
This
is
the
envoy
logo,
but
this
applies
to
to
all
service
measures.
We
put
a
proxy
alongside
each
one
of
these
applications
and
that
handles
the
network
traffic
for
us.
A
So
you
know
a
call
from
pod
a
to
pod
B.
Now,
if
Part
B
is
down,
then
the
Envoy
next
to
pod
a
can
say:
okay
I
will
find
another
instance
of
point
B
and
we'll
reach.
I
have
buffered
your
request
and
I
will
retry
it
and
I
will
I
will
get
you
the
answer
as
soon
as
I
get
a
correct
one
and
if
not
I
will
throw
a
circuit.
Breaker
and
I
will
return.
A
You,
the
last
answer
that
we
got
or
a
default
answer
or
something
like
that
so
service
a
can,
have
the
business
logic
it
had
before,
and
it
doesn't
have
to
worry
about
the
new
network
environment
that
it
is
in
because
that
is
complicated
and
it
can
fail
so
to
make
an
a
service
measure
useful.
We
need
a
way
to
configure
all
of
these
proxies.
You
know
they
need
to
do
different
things.
A
There's
no
side
effect
of
that
that
operation
is
called
idempotent,
so
each
of
these
proxies
yeah
and
you
might
want
a
different
time
out
for
different
routes.
You
might
want
all
kinds
of
different
configuration,
so
each
of
these
proxies
gives
you
all
of
these
features,
but
they
need
to
be.
The
point
of
this
talk
is
that
they
all
need
to
be
told
what
to
do,
and
that
depends
on
your
application.
A
So
a
service
motion
comes
with
a
control
plane.
Sto
has
a
control
plane
you're
under
CLI
sto
CTL,
which
you
use
to
talk
to
it
and
the
control
plan.
You
you
give
the
control
plane
a
simple
definition:
quite
simple
definition
of
wow:
you
want
the
service
mesh
to
act
and
then
the
control
plane
will
configure
all
of
those
proxies
for
you.
A
This
is
another
bad
joke.
I
would
say
it
is
not
at
all
plain
sailing
where
you,
if
you
think
everybody
knows
about
kubernetes
here,
if
you
think
about
how
you
use
Cooper
Nettie's,
when
you
first
start,
you
just
make
a
deployment
all
right
and
a
service
to
resources-
and
one
says
here
is
the
workloads.
I
would
like
you
to
run
here,
is
code
to
execute,
and
then
the
service
says-
and
here
is
how
you
expose
it
over
the
network.
A
But
if
you
want
to
make
that
application
resilient,
if
you
want
to
use
kubernetes
in
production,
then
you're
going
to
need
ingress
right,
because
your
product
owner
probably
wants
several
services
on
a
path
based
route
to
behind
the
same
IP.
You
want
a
horizontal
pod
autoscaler
that
gives
you
resilience
against
having
lots
of
users
having
lots
of
users
is
a
good
thing,
but
if
you
only
have
one
pod
and
lots
of
users
that
pod
will
crash,
so
you
need
to
be
resilient
against
that.
You
need
to
tell
kubernetes
how
to
scale
your
application.
A
You
need
to
set
requested
limits.
You
need
to
set
readiness
and
liveness
probes.
You
also
probably
need
some
config
maps
and
some
secrets
and
some
upgrade
strategies.
The
translators
could
ignore
this
I
have
a
ji
affinity,
anti
Finity,
pod
destruction,
budget,
pom,
security
policy,
Network
policy,
pod,
preset
and
service
account
in
our
back
limit
range
cluster
autoscaler.
There's
probably
more.
My
point.
Apologies
to
the
translator.
My
my
point
is
that
it's
not
as
simple
as
just
telling
kubernetes
to
run
your
workload.
A
You
need
to
to
configure
all
of
these
extra
things
to
get
the
most
out
of
it.
So
good
realities
can
do
some
stuff
out
of
the
box.
It
will
have
a
get
if
your
application
actually
crashes.
If
it
just
seg
faults,
then
kubernetes
nodes,
that's
bad,
then
it
will
restart
it,
but
there's
lots
of
other
failure
modes
like
it
just
you
know
it's
still
something
on
the
port,
but
it
just
never
replies
because
maybe
it's
deadlocked
kubernetes
doesn't
know
how
to
to
detect
that
in
the
general
case,
so
you
have
to
write
a
health
check.
A
You
know
a
liveness
probe
that
hits
the
HTTP
port
and
tries
to
get
a
response,
and
only
by
doing
that,
can
you
tell
kubernetes
that
you
know
this.
This
service
is
a
network
daemon.
That
is
meant
to
be
replying
to
HTTP
requests
and
if
it
doesn't
do
that,
then
it's
broken,
so
configuring
kubernetes
is
is
about
describing
your
application
to
kubernetes,
saying
hey.
A
We
have
this
unix
process,
it's
a
black
box
to
you,
but
I'm
going
to
tell
you
that
it
can
only
do
1,000
requests
per
second
and
when
it
starts
doing
that,
you
know
when
it
gets
to
1,001.
You
need
to
make
a
new
instance.
It's
listening
on
TCP.
So
if
you
send
a
valid
TCP
request
and
you
do
not
get
a
valid
TCP
response
or
you
get
a
th
TTP
response
that
says
500
in
it,
then
it's
broken.
A
So
as
I
say
we
use.
We
can
use
this
teo
when
we
have
one
of
these
distributed
systems
and
we
need
to
help
out
with
some
of
these
and
new
distributed
systems
failure
cases.
So
the
the
tag
line
says
that
sto
is
an
open
platform
to
connect
to
secure,
to
control
and
to
observe
micro
services,
and
all
of
these
words
are
actually
quite
important
and
I'll
I'll
go
through
them
in
this
talk,
but
basically
what
I?
What
this
talk
is
about
is
that
you
need
to
do
the
same
with
this.
A
A
I'm
not
going
to
send
my
demo
up.
I
have
a
require
can't
get
my
VPN
to
work.
I
have
a
recording
so
we'll
skip
that
I,
just
I
just
can't
so
so
the
observability
part.
This
is
going
to
be
really
awkward
because
I
can't
see
the
screen
so
yeah.
So
it's
I
can
do
all
of
this
stuff
and
I
guess
by
default.
It
will
connect
your
applications.
A
If
you
install
this
do
and
then
you
deploy
your
application,
the
sidecar
is
there
and
it
intercepts
all
of
the
traffic
and
it
moves
the
traffic
from
one
place
to
another.
So
pod,
a
it's!
A
pod
B
involves
that
the
traffic
going
through
two
copies
or
rhomboid
and
like
kubernetes
it
does
lowest-common-denominator
stuff.
If
you're
trying
to
talk
to
service
B,
then
it's
going
to
pick
the
traffic
up
and
it's
going
to
move
it
to
service
B.
A
Of
course
it
is,
but
it
won't
retry
it
by
default,
because
in
the
general
case,
that's
not
the
thing
to
do
in
the
general
case.
That's
not
safe,
because
it
might
not
be
an
ident
potent
operation.
You
also
can't
necessary.
You
might
think
you
want
to
secure
all
the
traffic
between
parts,
but
in
the
general
case
you
actually
can't
do
that
because
it
interacts
weakly
with
some
other
systems.
And
what
does
control
mean?
You
know
if
one
pod
breaks
do
I
want
to
route
traffic
around
it?
Do
I
want
to
return
a
default.
A
A
So
the
first
is
observability
observability
is,
is
three
main
parts,
its
metrics
about
the
traffic?
It's
a
service
graph
of
the
traffic
which
pod
talks,
to
which
you
know
when
you
had
a
model.
If
you
can
just
attach
a
debugger,
you
know
if
there's
a
problem
in
namespace
a
you
attach
a
debugger
and
you
step
through
when
you
find
out
that
it's
calling
such
as
some
code
in
namespace
C
and
that's
where
the
underlying
core
problem
is
the
root
cause
with
a
micro
service.
You
can't
do
that
service.
A
A
The
green
tick
is
me
saying
that
I
think
you
should
do
these
things
all
the
time
right
and
these
are
safe
and
great,
and
you
should
always
do
them
so
I
have
a
demo
I
hope
you
can
read
it
because
I
can't
make
this
terminal
any
bigger.
Now,
okay,
so
I
have
some
scripts
to
do
the
demo,
because
I
don't
want
to
type
I
already
have
a
gke
cluster
to
show
you
this
in
because
they
take
a
long
time
to
spit
up.
I
recorded
this
over
the
VPN
in
the
hotel,
so
the
latency
is.
A
It
was
deliberate,
I
promise,
so
we
have
a
one-load
kubernetes
cluster
I
get
and
I
have
downloaded
the
latest
Cisco
1.02
and
I
have
already
downloaded
helm
when
you
need
help
to
install
this.
Do
so.
Ok,
this
is
this.
Is
me
installing
SDM
into
the
cluster?
It's
normally
quite
fast,
but
the
latency,
the
latency,
was
bad.
A
What's
interesting
is
that
this,
this
installs
iste
ió,
so
stos,
is
some
pods
right.
Stos
is
some
workloads,
the
str
control
plane?
It
is
code
that
runs,
but
also
when
you
install
this,
do
it
installs
a
lot
of
configuration.
So
all
of
this
is
is
the
default
configuration
for
Sto,
it's
not
baked
into
the
code.
It's
supplied
in
kubernetes,
config
Maps.
You
know,
like
I,
said
at
the
start.
That
is
a
something
you
should
do
with
kubernetes.
A
That
is
a
best
practice,
is
to
have
your
configuration,
managed
separately
and
provided
through
config
Maps,
and
that's
exactly
what
sto
does
so.
This
is
kind
of
what
sto
does
out
of
the
box.
You
can
go
and
read
all
of
these
files,
they're
actually
quite
interesting,
so
under
the
Box,
it
does
a
lot
of
stuff
for
you
and
observability
is
one
of
those
things
which
I'll
show
you.
It
doesn't
do
everything
though
I.
A
If
I
was
hacking,
it
would
look
really
good,
but
I'm.
Just
standing
here
come
on.
It's
Deva
you
see.
Sto
knows
how
to
use
kubernetes
sto
is
installing
horizontal
pod,
auto
scalars,
to
describe
sto
control
plane
to
kubernetes,
so
kubernetes
know
how
to
keep
STR
running
so
that
sto
can
keep
your
application
running.
A
Self
abetik
all
order
right
t.
No,
it's
not
there.
We
go
there.
We
go
right,
so
history
is
installed.
I'd
forgotten,
I'd
recorded
that
now
going
to
deploy
the
book
info
application.
It's
a
sample,
micro-services
application.
If
you've
looked
at
this
Jo
or
the
documentation,
you've
probably
seen
it
I
didn't
run
my
own,
because
I
think
that
actually
does
confuse
things.
I'm,
not
lazy,
I!
Think!
A
There's
something
I
want
to
take
right,
so
you
can
see
with
all
the
all
the
pots
coming
up.
This
is
standard
Cuban,
it
is
ya,
know
file.
It
only
describes
it
only
talks
about
one
container
in
the
pod,
but
the
pod
can
have
more
than
one
container,
and
we
know
that
what
is
tio
does
is
put
that
side
car
next
to
our
application.
A
So
this
here,
0
of
2,
means
that
we're
still
waiting
for
this
to
start
that
we're
waiting
for
two
containers
to
start
and
one
is
the
application
which
is
in
the
pods
back
in
the
pot
llamo
and
the
second
one
is
the
envoy
side
car
that
we
as
a
user,
do
not
tell
kubernetes
about
when
we
describe
our
applications
in
kubernetes.
We
don't
describe
the
side
car
because
it's
not
part
of
the
application
is
still
injects
that
it's
joe
does
that
automatically.
A
A
This
is
book
info
if
everybody
knows
tells
a
website
telling
you
about
one
book
book
details
are
on
the
left.
Fake
reviews
of
the
book
are
on
the
right.
Some
of
the
reviews
have
we
keep
hitting
refresh
some
of
the
reviews,
have
this
star
rating
and
some
do
not
because
we
are
a
be
testing
three
different
versions
of
book
info.
One
shows
no
stars,
one
shows
them
in
black
and
one
shows
them
in
red.
A
So
we
just
hit
that
and
refreshed
it
a
whole
bunch
of
time.
So
we
generated
traffic
by
all
of
the
micro-services
work
together
and
gave
us
that
book
info
page
I
know
it's
ugly,
but
like
six
microservices
work
to
give
you
that
page
and
we
just
did
it
a
bunch
of
times
so
what
I'm
doing
now
is
I'm
just
setting
up
again
setting
up
a
port
forward.
It's
it's
boring
plumbing,
but
I'm
setting
up
a
port
forward
to
a
part
of
sto
called
key
alley,
and
that
is
an
observability
dashboard.
A
So
one
of
the
things
we
get
for
free
with
this
job.
Well,
one
of
the
things
that's
very
useful
in
running
a
resilient
application
is
observability
and
the
ability
to
get
deep
insights
into
your
application
and
how
it's
running
we
have
to
log
in
to
Kali,
and
now
we've
done
that
we
can
see
that
there
is
a
kubernetes
namespace
default
with
four
things
running
in
it
like
six
pods,
but
three
of
them
were
the
different
versions
of
the
star
rating
book
so
for
morale
applications
and
completely
for
free.
A
We
have
not
described
this
like
we
write
a
load
of
Yama
that
describes
our
application
to
kubernetes
and
will
soon
we'll
be
describing
it
to
ISTE.
Oh,
we
have
not
described
this
graph.
This
is
made
empirically
a
posterior
eye.
I,
don't
know,
what's
easy
to
translate.
This
is
made
from
the
traffic
that
is
being
sent
through
the
cluster.
It's
amazing.
This
is
a
simple
application,
but
you
see
the
graph
of
all
the
micro
services.
Netflix
House
right,
it's
huge!
It's
thousands,
so
you'd
be
amazed.
A
How
many
people
don't
know
this
about
their
own
set
of
micro
services,
so
first
step
two
to
getting
the
most
out
of
your
system
and
making
it
the
highest
quality
is
to
understand
it.
So
this
is
a
really
really
powerful
tool
to
to
understand
it,
which
is
why
I've
put
it
in
here
and
you
can
get
a
bunch
of
different
a
bunch
of
different
views
from
this
thing
running
a
little
bit
short
on
time.
So
I
will
skip
through
this.
A
Then
the
next
thing
you
want
to
do
to
get
your
resiliency
is
to
take
advantage
of
the
traffic
routing
in
sto.
This
happens
at
layer
seven.
So
this
understands
the
HTTP
protocol
and
can
do
more
advanced
things
because
it
understands
more
about
the
intent
of
the
transaction
because
it
can
understand
the
HTTP
header.
So
we
can
describe,
we
can
tell,
is
tell
about
the
different
versions
you
see.
There
were
three
versions
of
the
star
rating
micro
service.
Kubernetes
doesn't
know
anything
about
that.
A
They
all
share
the
same
labels,
so
we
just
round
robins
traffic
between
them
as
far
as
its
concerns
they
might
as
well
be
the
same
thing.
We
know
there
are
three
different
versions
of
we
know
we
should
be
treating
them
differently,
so
we
can
tell
is
deal
about
that.
We
can
then
start
routing
the
traffic,
so
we
can
pin
it
all
that
the
demo
will
show
you
that,
first
of
all,
we
will
say
the
a/b
test
is
not
ready.
A
Yet
you
know
we
have
deployed
version
2
and
version
3,
but
they're
only
internal,
so
we'll
send
all
the
traffic
diversion
one
and
then
we
can
start
shifting
it
to
to
version
2
and
version
3.
So
I
can
an
a/b
test.
Basically,
you
can
do
Canaries,
that's
a
popular
word,
but
really
that's
just
an
a/b
test
with
a
much
smaller
sample
as
far
as
I'm
concerned
and
there's
other
things.
This
do
can
do
for
you
because
we
have
this
powerful,
Envoy
proxy
in
our
network.
A
You
can
do
really
clever
things
like
if
your
toric,
if
one
application
is
serving
XML
and
another
application
wants
to
consume
Jason,
you
don't
have
to
translate
that
in
either
application
you
can
just
tell
envoy
to
do
it.
The
network
will
just
transparently
do
that.
So,
if
you
can
describe
to
sto,
hey
I
just
just
deployed
an
application,
but
it's
legacy.
It's
some
third
party
piece
of
software
I
bought
and
it
only
talks.
A
Xml
I
want
you
to
make
it
look
like
it
talks
Jason
to
all
of
my
other
services
right
and
that
avoids
you
writing
a
little
another
little
micro
service
that
tries
to
do
that.
Translation
and
sticking
it
in
the
middle,
because
you
will
probably
get
it
wrong,
though,
there's
bugs
in
all
code.
Why
would
you
light
that
one
that
already
exists?
So
that's
another
step
to
making
your
application
more
resilient,
I
think
the
demo
for
this
is
quite
good,
so
I'll
do
it
quickly.
A
A
However
many
times
you
hit
refresh,
you
always
get
version
2.
You
always
see
those
star
ratings,
but
that's
not
random
a/b
test.
That
says
that
you
know
Jason
works
for
us.
He
wants
early
access
to
v2
he's
going
to
hammer
that
before
we
let
it
use
loose
to
the
users
and
Jason
will
get
v2.
Everybody
else
will
get
v1,
but
that's
based
on
looking
at
the
HTTP
header.
That
says
who
is
logged
in
right,
so
you
can
take.
You
can
do
much
more
clever
things
like
that.
A
You
can
send
a
new
version
just
to
the
users
in
France.
You
can
send
a
new
version
just
to
the
users
who
are
using
alpha
browsers
because
you
think
they're,
the
kind
of
people
who
like
to
take
risks
right
and
we'll
maybe
brain
blame
their
browser.
If
your
website
goes
wrong,
I
mean
that's,
that's
great
I
think
we
don't
have
time
for
the
next
demo.
I,
don't
think,
but
that's
showing
you
our
back,
so
we
can
add
it.
We
can
get
this
year
to
add
actual
resiliency
features.
A
So
there's
a
few
things
we
might
want.
We
might
want
a
timeout
say
one
application
is
being
slow.
This
is
another
cascading
failure
right
everything
blocks
waiting
for
it.
Obviously
you
can
write
asynchronous
code,
but
no
everything
will
block
waiting
for
it.
So
we
can
tell
this
do
that
if
a
back-end
doesn't
reply
in
two
seconds
it
should,
it
should
just
send
a
500
as
if
the
other
end
had
done
that
right,
because
the
other
service
is
in
a
deadlock
and
it's
just
going
to
sit
there
forever.
A
So
we
can
tell
this
yo
to
just
give
up
on
your
behalf
right.
This
is
now
code
that
you
don't
have
to
write
in
your
micro-services
because
it
wasn't
there
because
we
remember,
we
have
that
monolith
and
a
call
from
you
know
name
space
8
in
the
in
space.
Seek
it
never
failed.
It
wouldn't
have
probably
never
wait
that
long
either
the
whole
thing
is
crashed
or
it
hasn't
now.
We've
introduced
this
network
that
might
be
super,
slow
and
super
congested
might
be
on
the
other
side
of
the
world.
A
We
can
also
have
circuit
breakers,
which
are
more
brutal
version
of
the
same
thing
and
say
that
if
a
service
fails
three
times
within
a
certain
window,
we
just
we
just
decide
that
it's
dead.
We
just
never
talk
to
it
again.
That
makes
things
resilient
if
a
service
is
being
flaky,
maybe
the
underlying
hardware
is
failing.
Then
we
can
just
do
that.
As
I
said.
At
the
start,
we
can
get
sto
to
do
retries
for
us
and
that's
very
configurable,
so
we
can
tell
it
only
certain
things
should
be
retried.
A
We
can
tell
it
how
often
how
fast
to
try
we
can
rate
limit
things.
This
is
really
important
for
making
your
system
resilient.
You
know,
no
software
is
free
of
bugs,
and
no
software
is
infinitely
scalable.
So
you
should
test
your
software.
You
should
know
how
many
requests
a
second,
your
micro
service
can
cope
with
and
still
still
meet.
Your
SLO
so
still
say
return
99%
of
answers
within
100,
milliseconds
or
whatever
is
susceptible,
and
then
you
find
out
that
that
limit
for
this
release
is
1,200
RPS,
and
then
you
set
a
rate
limit.
A
You
say:
hey,
sto,
don't
ever
learn
more
than
1,200
RPS
hit
this,
so
that
again
is
another
key
to
resiliency
and
you
you
would
also.
You
would
then
describe
that
to
kubernetes
as
well,
so
that
maybe,
when
you
to
a
thousand,
you
would
start
to
spit
up
more
copies
of
the
party
in
anticipation,
a
counterintuitive
thing
you
can
do
to
increase.
Resiliency
is
actually
to
induce
introduced
faults,
so
sto
can
introduce
delays
and
it
can
introduce
faults.
Sorry,
like
HTTP,
five
hundreds,
but
this
is
this
is
so.
This
is
chaos
testing.
A
If
anybody's
heard
of
chaos,
queue
or
chaos
monkey
these
these
things
come
along
and
they
kill
AWS
instance
it's
right
and
they
kill
pods
at
random
and
they
make
sure
that
your
you
know
you
run
a
test.
While
that
is
happening,
and
then
you
know
that
your
application
can
cope
with
it
and
it's
resilient
to
it
right
resilient
to
compute.
Failing
now
we
have
distributed
our
application
and
it's
running
all
of
these
calls
are
across
a
network.
We
have
another
failure
mode,
the
network
might
fail.
So
what
we
do
is
we're
in
staging.
A
Certainly,
we
get
SEO
to
pretend
that
that
network
is
slow
or
to
pretend
that
that
network
is
unreliable,
and
then
we
know
that
our
application
we
can
test
whether
our
application
still
copes
with
that
either
through
code
in
the
application
or
through
other
things
like
rejoice
and
timeouts
that
we've
configured
it's
yet
to
do
so.
That
may
seem
counterintuitive.
There
is
an
argument
that
you
should
do
this
in
prod.
You
should
test
in
prod,
see
charity
majors
talks.
There
is
an
argument.
A
You
should
have
this
running
all
the
time,
but
definitely
in
your
staging
environment,
where
you're
also
using
you
know,
doing
soaked,
ESTs
and
using
charlo
generators
and
all
that
kind
of
stuff.
You
should
have
this
kind
of
chaos
running
and
as
I
say
it's
counterintuitive,
but
it
does
increase
your
resiliency
and
it's
a
service
that
sto
can
offer
security.
This
is
your
resiliency
against
being
attacked.
You
know
your.
A
If
you
miss
configure
things
a
the
internal
part
of
your
application
may
be
the
actual
by
no
bank
account
records
part
would
only
have
been
called
by
other
things.
You
know
only
half
not
would
not
have
had
an
API
directly
on
it.
We
accessed,
you
know
through
other
parts
of
the
monolith
and
I
may
be
doing
authentication.
That's
not
good
enough.
Now,
because
there
is
this
thing:
we'll
have
to
listen
on
an
API
of
its
own.
A
So
if
somebody
can
break
into
any
pod
any
other
part
of
your
infrastructure,
they
now
have
access
to
that
in
the
way
that
they
didn't
before.
So
you
want
mutual
TLS
for
encryption.
You
want
strong
authentication
to
know
which
services
you're
talking
to
and
you
can
then
turn
on
service
authorization
which
says
that
if
I
am
the
bank
account
records,
only
the
UI
component
can
talk
to
me
because
it
displays
balances
and
that's
fine,
but
the
the
micro
service
I've
got
that
on
those
new
users
should
never
need
to
see
anybody's
bank
account
records.
A
A
That
was
what
the
last
time
I
was,
but
I've
kind
of
I've
kind
of
run
out
of
time,
and
then
your
other
things
that
you
just
don't
want
to
re-implement
or
that
you
don't
want
to
bother
using
a
library
for
in
every
application
things
like
injecting
cause
headers
any
other
kind
of
security,
middleware,
JWT,
validation,
xxs
mitigation,
any
kind
of
web
application
firewall
function,
you
can
have
sto.
Do
you
just
describe
your
application?
You
say
you
can
see
that
this
thing
is
listening
on
port
80.
A
Yes,
it's
talking
the
HTTP
protocol
and
this
isn't
a
REST
API.
It's
actually
talking
that
to
a
browser,
so
I
want
you
to
do
order.
The
standard
stuff,
like
you
know,
injection
attack,
detection,
and
course
header
injection.
So
this
just
means
that
you
don't
have
to
download
a
library
into
your
application
to
do
that.
Hooking
into
your
build
system
check
that
it's
working
properly
upgraded
every
time,
there's
a
patch
you
just
let
the
network.
Do
that
and
you
know
I
say
all
of
those
things
about.
It's
been
it's
complicated
to
have
your
app.
A
Do
it,
of
course,
a
lot
of
people?
Just
don't
do
it
at
all,
because
it's
too
hard,
but
with
oh
yeah,
one
yeah
more
file
now
sto
will
do
that
for
you.
So
sto
has
a
lot
of
power.
I!
Think
the
demos
that
you
see
are
normally
around
the
clever
routing
that
you
can
do,
and
that's
it's
great,
it's
great
to
have
those
features,
but
the
network
is
now
a
really
important
part
of
your
application.
A
It
has
a
really
important
effect
on
how
all
of
the
different
micro
services
work
together
as
one
unit
and
that
one
unit
has
to
work
as
one
thing
so
that
your
users
have
a
good
experience
right.
So
in
the
same
way
that
you
need
to
tell
kubernetes
a
lot
more
than
just
please
run
this
pod.
You
need
to
tell
this
do
a
lot
more
than
please
just
move
my
traffic,
because
the
network
was
doing
that
for
me.
A
If
it
just
moves
the
traffic
you
get
observability
for
free,
which
is
a
really
good
step
towards
a
resilient
system,
because
if
you
can
observe
it,
you
could
control
it
and
understand
it,
but
also
you
can
configure
sto
to
do
the
reach
eyes
and
the
timeouts
and
everything
else
that
we've
that
we've
seen
some
of
the
things
I
went
quite
fast.
Some
of
the
items
had
like
a
yellow
question
mark,
which
meant
sometimes
you
need
this.
A
So
a
retry
is
not
always
safe
and
some
things
had
a
Red
Cross,
which
actually
means
you
really
only
want
to
do
this
in
an
emergency.
So
you
can
do
things
like
Laura
love
the
traffic
and
there
are
privacy
implications
to
that
security
implications
to
that.
But
if
something
is
going
really
wrong,
you
can
get
SEO
to
keep.
You
know,
keep
the
service
up,
keep
sending
traffic
to
the
production
instance,
but
also
send
a
copy
of
that
traffic
to
your
laptop.
So
you
can
read
it
and
try
to
work
out.
A
A
That's
a
really
good
question,
so
the
question
was
what,
if
you're
not
running
in
kubernetes
sto
by
the
way,
runs
on
like
console
amazed
off
the
older
staff,
but
if
you're
running
in
server
less
a
CI
virtual
cubelet,
then
the
newer
systems
of
what
do
we
do?
I
guess
what
I
would
I
think
you
do
want
the
same
features
and
the
same
technology
is
actually
giving
you
that
soak
a
native
serve
is
a
server
this
framework
or
I
take
it
can
scale
to
zero
and
all
of
that
stuff.
A
It's
based
on
Sto,
so
two
things
that
are
on
K
Knights
you
serve
even
though
their
server
lists
you
don't
have
to
deal
with
the
deployments
or
the
rolling
upgrades,
because
it's
a
server,
that's
framework.
The
the
traffic
between
them
is
still
going
through
history,
okay,
native
server
installs
this
Joe
and
you
just
don't
see
it
so
we're
already
like
one
layer
of
abstraction
higher
than
this.
The
other
thing
is
that
Envoy,
the
proxy
used
by
sto
just
last
week
released
like
a
library
version
for
iOS
and
Android.
A
You
can
at
least
have
the
client
do
timeouts
and
retries,
because
you
can
now
download
air
void
as
a
library,
and
you
can
link
it
against
your
Android
application
and
even
if
the
server
as
a
whole
just
fails
and
says
500,
you
can
get
me
the
Android
application
to
retry
and
then
to
time
it
out,
because
envoy
is
now
being
released
as
a
life.
So
does
that
help
it's
more
about
gear?
It's
about
what
you
have
to
do
at
the
end
of
the
day
is
get
those
air
voids
in
there.
A
Next
all
of
the
components
in
your
system-
yes,
I!
Guess,
if
you're
using
a
virtual
cubelet
I,
don't
know
how
you
do
if
you're
using
faregates
virtual
cubelet
I,
don't
know
how
you
do
that
because
they
take
care
of
the
computer
fabric
and
you
don't
have
enough
control
to
inject
envoi,
but
a
surprising
amount
of
the
systems
that
that
do
present.
That
higher
level
of
abstraction,
like
okay
native
serve,
actually
have
this
stuff
built
in
and
will
do
it
for
you
and
in
the
worst
case,
you
just
have
to
client.
Do
it?