►
From YouTube: Living with the Pathology of the Cloud: How AWS Runs Lots of Clusters - Micah Hausler, Amazon
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Living with the Pathology of the Cloud: How AWS Runs Lots of Clusters - Micah Hausler, Amazon
Disk speed screeches to a crawl, packets get dropped, connections time out: welcome to the cloud! Most of the time the cloud "just works", but when it doesn’t, how does Kubernetes and etcd handle failure? In this talk Micah will discuss considerations for building and configuring cloud native systems for failure including how Amazon EKS’s architecture and design accounts for outages and dependency failures. Micah will also cover and lessons learned from managing lots and lots of Kubernetes and etcd for customers around the world.
https://sched.co/Uaav
A
So
welcome
to
San
Diego
Cuba
knots,
I've
not
been
back
to
San
Diego
since
I
was
actually
born,
was
born
in
the
area,
but
I've
never
been
back.
So
this
talk
is
living
with
the
pathology
of
the
cloud
and
how
AWS
ruins
lots
and
lots
of
clusters
I
couldn't
fit
in
lots
and
lots.
I
think
there's
like
a
75
character
limit,
so
I
had
to
abbreviate
it
we'll
jump
right
in
so
first
I
kind
of
wanna
give
a
little
bit
of
background
on
Amazon
eks.
It's
our
elastic
kubernetes
service.
You
may
have
heard
of
it.
A
You
don't
have
to
follow
all
of
what
we
do
to
build
a
planet-sized
line
at
scale
application,
but
I
kind
of
want
to
pull
back
the
curtain
a
little
bit,
and
so
you
can
kind
of
see
how
does
AWS
run
a
service
and
or
build
a
service
and
kind
of
talk
about
some
design
decisions,
trade-offs,
nuances,
so
there's
not
always
a
best
answer
to
everything.
So
what
are
some
of
the
decisions
we
made
for
operating
our
service
so
before
I
even
jump
into
the
service
itself?
The
first
thing
I
wanted
set.
A
First,
just
some
level
setting
is
what
are
the
eks
service
priorities
so
at
Amazon?
You
may
have
heard
this
if
you've
heard
other
Amazon
talks
talking
about
services,
the
first
really
two
things
that
we
really
value
in
operating
our
services,
our
security
and
operational
reliability.
These
aren't
just
things.
We
say
we
actually
like
say
these
to
each
other,
a
lot
day
to
day
when
we
get
like
a
security
issue,
whether
it's
a
CBE
or
just
something
that
we
noticed
or
something
that's
even
a
concern
of
a
customer.
A
We
take
that,
like
very
seriously
that
good,
like
gets
escalated
very
very
quickly
and
we
drop
everything
to
make
that
right,
and
that
goes
the
same
for
our
operational
reliability
as
well,
like
our
customers,
trust
us
to
run.
You
I'm,
assuming
you
here,
are
some
of
our
customers
to
run
your
critical
applications
on
our
service,
and
so
we
take
that
really
seriously
and
we
want
to
offer
a
really
highly
secure
and
stable
environment
for
you
to
do
that.
A
That's
that's
kind
of
the
level
that
we
operate
and
we
take
again.
We
take
like
security
and
operational
stability
very
very
seriously.
We
want
you
to
basically
rely
on
us
to
make
sure
that
the
cluster
work,
the
control,
plane
works
and
your
applications
run
so
backing
up,
there's
kind
of
a
high-level
overview
of
what
does
the
sort
of
service
map
look
like
for
when
you
have
an
e
KS
cluster.
So
what
is
it?
What
is
AWS
provide
for
you
today?
It's
really
well
up
until
yesterday,
it
really
was
just
the
managed
kubernetes
control
plane.
A
So
what
does
that
mean?
Well,
if
you're
running
kubernetes
there's
the
sort
of
three
primary
components,
the
cube
root
cube
api
server,
the
controller
manager
is
scheduler
and
then
there's
the
sed
database
behind
it.
It
CDs
the
key
value
store
database.
That's
open
source,
we'll
get
into
the
a
little
bit
more
of
a
TD
later,
but
we
run
that
across
multiple
availability
zones.
For
you
for
high
availability.
We
automatically
fail
away
from
any.
You
know
AZ
in
a
specific
zone
issue.
A
If
there's
an
issue
aspect
affecting
a
specific
zone,
we
maintain
that
these
actually
implement
this
with
Auto
scale
groups,
just
like
you
would
as
a
customer
for
your
own
application
using
a
lot
of
the
same.
A
Tobias
primitives
we'll
get
into
a
little
bit.
We
have
a
network
load
balancer
again,
another
AWS
product,
surprise
in
front
of
the
API
servers,
and
then
you,
the
customer,
have
the
data
playing
what
we're
calling
in
your
account.
A
So
that's
where
you
see
your
ec2
instances
and
those
can
be
you
see,
join
Tunes
instances
you
launch
in
the
console
could
be.
It
could
be
more
likely,
probably
auto-scaling
groups
that
you
manage
with
either
cloud
formation
or
terraform
or
something
like
that
or
cops
or
any
or
not.
Guess
not
cops
in
this
case.
But
cops
is
this
sort
of
similar
thing
for
it's.
A
It's
a
data
plane,
so
a
feature
we
just
launched
yesterday
has
managed
node
groups,
and
you
see
that
here
that
feature
I
won't
go
into
a
huge
feature
pitch,
but
what
the
basic
premise
is
that
you
can
click
a
button
and
get
nodes
and
get
minute
upgrades,
so
you
can
look
for
more
information
there,
but
back
to
kind
of
our
service.
Well,
how
do
we
build
this?
Well,
as
I've
been
saying
it's
all,
based
on
all
these
AWS
primitives
we're
not
using
any
super-secret.
B
A
There's
a
few
api's
pop.
You
know
like
things
like
billing
that
I
would
we
expose
that
as
an
API?
You
don't
need
that,
but
for
the
for
the
rest
of
the
service,
it's
all
public
API
s,
one
of
the
other
things
that
we
do
is.
As
we've
built
the
services
we've
adopted.
What's
called
a
cellular
base
architecture.
This
is
becoming
more
and
more
popular
inside
AWS
and
we're
starting
to
talk
about
this
more.
What
this
means
is.
A
A
A
Like
we
have,
we
operated
a
really
large
scale,
we
have
small
high-functioning
service
teams
and
they
can't
do
one-off
things
for
every
customer
that
we
we
operate.
So
we
we
really
need
high
skyper
formance
high
scale
operation
here.
So
this
this
horizontal
starting
sort
of
helps
us
scale
out
not
up,
especially
when
you
do
operate
at
scale
you,
you
typically
do
hit
lines
not
and
I,
don't
just
mean
AWS
limits.
These
can
be
physical
limits.
A
All
right
like
how
many,
how
many
packets
can
you
get
in
a
in
a
network
connection,
or
you
know
how
much
bandwidth
or
how
much
CPU
you
can
get
on
a
box
like
yeah.
There
are
a
number
of
limits
that
you
start
to
hit.
So
that's
why
we
favor
more
the
the
scaling
out
model
other
benefits
to
this.
The
cellular
architecture.
Are
you
get
reduced
blast
radius?
A
If,
if
one
cell
is
affected
by
say
a
downstream
service
or
configure
8
miss
configuration
or
something
like
that,
you
don't
end
up
affecting
everything
in
that
will
call
it
partition
or
that
group
of
cells
we
actually
so
we
we
char
de
cells
by
region.
So
ETS
is
a
regional
service,
so
you
see
kis
in
u.s.,
East,
One
or
us
os/2
or
or
Dublin
or
some
other
AWS
region,
but
we
have
in
some
of
our
bigger
regions
up
to
20
cells
or
more
than
20
cells.
A
A
And
then
some
of
the
trade-offs
of
this
are
again
not
everything
is
PG
right.
There's,
there's
work
for
this,
there's
a
increased
complexity
if
I
have
a
say
a
customer
cluster,
we'll
call
it
a
customer
right.
Cluster
ID,
a
unique
identifier
for
a
cluster
and
I'm
an
operator
and
I
need
to
debug
an
issue.
I
can't
just
look
up
from
a
single
database
Oh
which,
which
cluster
is
this
I,
have
to
do
a
little
bit
more
indirection,
because
we're
split
in
two
separate
shards.
A
This
also
necessitates,
then,
the
upfront
investment
in
operational,
tooling
and
automation,
because
when
you
have
n
copies
of
something
instead
of
just
one,
if
you've
ever
op
like
for
those
of
you
who
are
AWS
customers
which
I
imagine
a
large
portion
of
you
are
here.
If
you
have
just
one
Amazon
account
like
no,
no
shade,
that's
fine,
but
what
I?
What
we
highly
recommend
is
is
splitting
up
your
if
you
have
a
pre-production
environment,
put
that
in
a
different
account,
because
that
helps
for
some
of
these
same
sorts
of
reasons
insulate
you
so
this.
A
If
you've
worked
across
multiple
Amazon
account,
so
you
sort
of
have
started.
If
you
see
the
need
for
okay,
I
need
tooling,
because
I
can't
just
query
it
in
in
one
account.
Imagine
that
times
20
per
region,
so
you
have
whatever
how
many
public
regions
are
there
18
now
I
think
and
you
have
numerous
cells,
we're
talking
about
hundreds
of
cells,
so
investigating
a
specific
issue
requires
an
investment
upfront
investment
in
in
this
automation
and
tooling.
A
So
how
is
the
cluster
created
you?
As
a
customer
say,
okay
of
your
command
line,
the
AWS
CLI,
you
say:
aw,
F,
CKS,
create
cluster
or
maybe
use
ETS
Kittel
or
you
use
the
console
or
whatever
is
well
you
in
the
end
or
the
customer,
making
an
API
request
against
an
API,
and
in
our
case
that's
it
was
on
API
gateway.
We
actually
implement
our
service
with
lambda,
so
we
have
a
lambda
backed
API
gateway
back
by
a
lambda
function
which
will
take
your
request.
A
Do
some
look
at
lookups
about
your
account
about
your
you
know:
authorization
authentication,
all
that
and
then
can
we
create
this
cluster.
Okay,
let's
create
this
cluster
and
this
gets
handed
off
to
through
a
number
of
number
of
steps,
but
it
goes
to
a
step
function,
so
we're
using
again
same
sort
of
a
public
AWS
tooling
that
you're
and
products
that
you're
familiar
with
and
in
this
whole
cellular
based
architecture.
A
A
So
when
this
step
function
gets
your
cluster
require
create
request.
The
next
thing
that
it
does
is
profit
obviously
process
this
request
and
creates
a
cloud
formation
stack
for
your
cluster.
So
in
this
setup
we
create
a
unique
stack
for
your
cluster,
which
contains
all
the
other
AWS
components
required
for
that
specific
cluster.
A
So
this
is
this
cluster.
How
is
it
created?
So,
like
we
said,
there's
a
step
function
creates
cloud
formation,
but
what
else?
All
in
that
cloud
formation?
Well,
there's
a
number
of
things.
So
it's
it's
a
progression,
and
this
is
sort
of
why
you
see
a
lot
of
question
you
get
is
like.
Why
does
the
eks
cluster
take
so
long
to
create?
So
today,
we've
just
it's
something
that
we
are
continually
optimizing
and
we'll
be
optimizing
more,
but
we
create
you
a
specific
VPC
for
your
cluster,
so
each
cluster
gets
its
own
AWS
PPC.
A
We
create
two
Auto
scale
groups,
one
for
at
cd14,
API
servers.
Then
I
am
roles
for
those
and
elastic
load,
balancers
in
front,
as
well
as
a
number
of
other
resources.
It
came
as
keys
all
those
kinds
of
things
so
that
that's
sort
of
the
like
the
high
level
stack
of
when,
when
your
clusters
created
these.
These
are
all
the
things
that
are
going
on
in
the
background
so
yeah
again,
this
is
just
kind
of
a
high-level
view.
A
So
now
now
that
the
cluster
is
created,
let's
talk
about
some
of
the
like
more
fun
stuff,
the
failure
stories.
So
in
creating
these
millions
of
clusters,
there's
a
lot
of
things
that
can
can
happen
right
most
of
the
time.
The
fast
fast,
fast
majority
of
timing,
creating
millions
of
Luster's,
nothing
bad
happens,
which
is
really
great,
but
even
when
creating
or
when
it's
up,
occasionally
things
don't
go
as
expected,
and
sometimes
this
is
a
downstream
dependency.
Sometimes
this
is
could
be
an
AWS.
A
You
know
you
guess
code
bug
it
could,
but
also
it
could
be
specific
to
kubernetes,
so
I
kind
of
want
to
dive
into
some
of
the
interest
in
kubernetes
learnings,
because
I
think
those
are
really
like
really
helpful
just
in
general.
So
one
of
one
of
the
more
interesting
ones
to
me
is
this:
this
semi
recent
bug
that
we
got
a
bug
request
from
a
customer.
It's
basically
queue.
Baby
GI
server
fails
to
connect
to
a
new
web
app
on.
A
The
ec2
node
that
the
pod
for
this
web
hook
was
running
on,
gets
terminated,
there's
no
fin
in
the
TCP
stream.
A
new
web
hook.
Pod
comes
up
on
a
new
instance
to
answer
requests,
but
the
API
server,
the
eks
managed
API
server,
doesn't
connect
to
this
new
web
hook
for
almost
15
minutes.
So
if
you're
familiar
with
this
to
you
at
all,
if
this
customer
is
running
when
you
don't
mutate
pods,
they
don't
get
the
Envoy
sidecar,
which
they
can't
connect
to
your
services,
which
is
production
outage.
So
that's
a
bad
thing
right.
A
A
So
when
you
actually
enable
cluster
logging
on
your
account,
that's
CloudWatch
log
agent,
putting
your
your
logs
into
carwash
logs,
one
of
the
other
things
that
we
do
when
we
create
these
instances
is
we
have
a
secondary
elastic
network
interface
or
a
and
I
attached
to
it?
Normally,
when
you
have
any
c2
instance,
you
see
there's
a
network
interface
attached.
You
can
attach
secondary
interfaces.
Well,
we
have
attached
one,
but
we
attach
it
in
your
account.
It's
across
the
county
and
I.
This
is
what
enables
cube
cuddle,
exec
and
Long's.
A
So
when
I
want
to
do
a
port
forward
or
a
logs
or
exact
to
get
down
into
my
cluster,
don't
with
kubernetes
that's
one
of
the
beautiful
things
for
deep
free
debugging,
you
don't
have
to
use
SSH
everywhere
to
get
into
your
pods.
How
does
that
work?
Well,
you,
the
client,
talk
to
the
API
server.
The
API
server
reaches
out
over
this
network
interface
into
you.
A
The
customers,
norther
customers,
CPC
to
talk
to
cubelet,
to
exact
logs,
do
whatever
it
needs
to
so
this
sort
of
same
communication
path
is
what
api
server
uses
to
talk
to
web
hooks
mutating
Webfoot,
mutating
admission
controller
web
hooks
in
your
cluster,
so
the
API
server
was
reaching
out
into
the
customers.
Account
like
normal
to
this
web
hook.
Pod
as
it
was
getting
pot
of
great
requests,
would
talk
to
this
web
hook.
Say:
okay!
Is
everything
good,
okay,
pass
it
on
mutate?
A
It
the
pod
gets
created,
except
in
this
case
the
node,
like
we
said,
went
away
for
whatever
reason,
and
the
problem
was
that
this
TCP
connection,
TV,
TCP
stream
or
or
this
connection
was
never
remade
to
the
new
web.
App
odd
because
the
kubernetes
scheduler
placed
it
on
the
write
that
a
new
one
on
the
rectangular
on
different
node
in
the
correct
place,
but
again
15
minutes
after
15
minutes.
Finally,
the
API
server
started
connected
to
the
new
one.
So
what
do
we
do?
A
We,
one
of
the
first
things
we
looked
at
was
okay,
go
into
the
go
to
the
API
server
run
netstat,
what
what's
actually
where's
this
connection
going
and
we'd
see
this
foreign
address
it.
So
the
the
local
address
cube,
API
server.
The
foreign
address
would
be
the
dead
nodes
IP,
but
it
still
shows
has
established
that's
weird
right
like
what.
Why
is
that?
A
Still
there
after
we
did
a
bit
of
digging
around
looking
to
sort
of
the
the
Linux
kernel
system,
tuning
parameters,
so
there's
this
TCP
tries
to
I
won't
read
all
this
text,
but
the
long
and
short
of
this
means
based
on
an
unacknowledged,
TCP
connection
because
well
I'll
back
up
a
little
bit
go
in
the
cube.
Api
server
is
using
obviously
TCP
here,
but
it's
also
using
HTTP
too.
A
So
the
go
net
HTTP
library
is
using
an
HTTP
to
stack
to
talk
to
the
mutating
web
book,
which
is
the
server
and
when
that
fin
is
never
sent.
The
TCP
connection
is
held
open
up
to
15
retries
here,
which
comes
to
that
15-minute
mark.
So
it
was
in
effect,
there
was
no
fin
being
sent,
but
goes
HTTP
to
stack,
has
a
bug
that
forced
that
connection
to
not
be
open.
Basically,
the
client
was
not
sending
a
ping
to
say
server.
Are
you
still
around,
and
so
it
just
kept
the
connection
open.
A
So
there's
actually
been
upstream,
go
a
bug,
report
and
and
patch
made
for
this-
that's
not
quite
yet
merged,
and
so
one
of
the
short-term
fix
that
we
actually
recommended
this
customer
in
this
case
is
on
your
Envoy
web
hook,
proxy
disable
ation
to
be
to
so.
Have
it
just
an
HTTP
1
connection?
Ok,
yes,
you
might
have
a
few
more
TCP
connections,
but
you
won't
have
this
15
knowledge
and
then
also
the
upstream
communities.
A
There's
a
upstream
change
until
the
NGO
patch
merges
and
then
the
kubernetes
upgrades
to
next
go
version
to
force
the
API
server
to
reach
out
over
HTTP
one,
even
if
the
the
web
book
server
offers
HTTP
to
one
of
the
other
interesting
edge
cases
I
think
is
at
CD.
So
we
did
this
earlier
about
how
STDs
a
managed
service
or
part
of
the
managed
service
for
you.
You
don't
actually
you're
not
actually
directly
exposed
to
it,
but
it's
the
open-source
key
value
data
store.
A
It's
based
on
the
raft,
consensus
algorithm
and
on
a
high
level,
but
I
won't
get
all
into
what
raft
is,
but
basically
wrapped
relies
on
what
we
call
consensus
so
more
than
or
some
number
of
the
instances
in
the
the
cluster,
the
sed
cluster
agreeing
on
something.
So
that's
how
state
is
maintained
when
you
make
a
change
in
kubernetes,
it
gets
propagated
down
into
its
V
and
the
the
cluster
agrees.
A
You
want
odd
numbers
for
this
corn,
not
even
numbers,
because
if
you
have
a
network
partition,
you
want
your
cluster
to
continue
to
function,
or
at
least
part
of
the
cluster,
to
continue
to
function
and
to
work
right.
So
in
this
case
or
we'll
use
the
example
of
five,
if
there
was
a
direct
network
partition
between
the
three
two
of
the
two
and
three
of
these
five
instances,
you
might
see
this.
These
sort
of
access
patterns
disappear.
A
The
smaller
left
side
of
the
cluster
would
not
have
porn
because
quorum
a
net
CD
is
more
than
half
n
minus
two
plus
one.
We
don't
meet
that
here
on
the
left
side,
so
that
if
you
try
to
do
reads
or
writes
for
those
nodes
from
kubernetes
api
server,
it's
gonna
fail,
but
on
the
right
side,
three
out
of
the
five
nodes
are
still
talking,
so
they
can
still
continue
operation.
So
this
is
actually
really
awesome
for
high
availability.
A
Hardly
ever,
though,
interestingly
I
mean
that
this
is.
This
is
very
helpful
and
good.
Or
is
there
a
direct
split
down
the
middle
like?
Sometimes
it's
one
direction
can
talk
about
the
other
can't
or
three
out
of
the
five
can
talk
to
each
other,
but
one
can
only
talk
to
one
of
the
others
and
it
gets
really
interesting.
A
But
for
the
simple
case
we'll
just
talk
about
this
so
kind
of
back
to
the
sort
of
a
3-node
case,
because
that's
that's
what
he
case
operates
today,
but
for
every
eks
cluster
that
you
create
or
that
we
that
we
create
for
you
there's
three
nodes.
So
what
happens
if
one
of
them
goes
away?
Well,
we're
still
good
at
quorum
right
like
we're
over
half.
A
But
what,
if
that
one
that
you
thought
was
gone,
is
now
actually
gone.
So
if
one
is
still
around
and
then
it
comes
back,
but
you
created
a
new
one,
and
now
you
have
for
your
quorum
just
changed.
So
in
this
case,
what's
quorum,
it's
half
plus
one.
So
that's
three!
So
now
that
we're
at
at
four
nodes
and
if
we
get
a
network
partition,
we're
not
in
a
good
space
right
like
we're
gonna,
we
don't
want
to.
A
We
want
to
avoid
slug
rain,
so
none
of
the
nodes
can
answer,
requests,
reads
or
writes,
and
that's
obviously
the
least
ideal
scenario.
So
one
of
the
the
interesting
cases
that
we've
seen
with
sed
that
we
experience
sort
of
early
on
was
around
this
sort
of
this
idea
of
nodes
that
you
think
are
gone
but
aren't
normally.
When
you
terminate
an
instance,
it
happens
almost
immediately,
but
on
a
very
like
single
digit
number
of
times
it
can
take
like
upwards
of
four
us.
We
found.
A
We
saw
a
weird
event
where
it
took
upwards
of
twenty
minutes
and
we
just
so
happened
to
have
the
same
habit
to
have
a
20
minute
time
out
in
our
sed
who
have
a
sed
manager
code.
So
we
terminate
this
instance
thinking.
It's
gone
our
nanny
code.
What
it
does
is
watches
for
that
member
to
delete
and
then
once
it's
actually
deleted,
removes
it
from
the
membership
of
the
cluster.
A
So
if
you
delete
an
instance
but
don't
go
to
the
sed
API
and
say
hey
that
guy's
gone
now,
the
cluster
still
thinks
he's
around
somewhere
because
it
doesn't
know
any
better,
and
so
in
this
case
this
is
sort
of
what
this
is.
What
happened
is
we
said:
okay,
we
have
now
a
new
instance
that
came
online
because
that
one
got
terminated
but
because
that
member
and
eventually
that
member
did
actually
terminate,
but
because
of
our
timeouts,
and
we
never
actually
just
cleaned
him
up
what
happens
next?
A
Well,
we
do
our
normal
say
you
a
cluster
upgrading,
so
we
also
upgrade
your
@cd
for
you.
It
happens
behind
the
covers,
but
we
are
automation,
thinking,
okay,
we
have
three
nodes,
we're
good.
If
we
terminate
one
to
do
our
normal
cluster
upgrade
everything's
going
to
be
fine
right,
we're
at
quorum,
we
think,
but
that
node
was
actually
never
terminated.
So
this
case
looks
almost
the
exact
same
to
at
CD
as
two
and
two,
because
now
two
instances
don't
have
a
quorum
out
of
four
so
that
that
was
about
to
me.
A
A
So
when
you,
when
you
think
that
you
have
remembers,
makes
like
if
you
did,
if
you're
doing,
if
you're
managing
it
see
to
yourself,
if
you
look
at
how
many
instances
you
have-
and
you
think
oh
I
only
have
three
instances
and
you
see
two
check
out
CD-
make
sure
they
actually
have
three
instances
in
your
cluster
and
then
check
your
check.
Your
depends.
These
are
telling
the
truth,
and
if
they're
not
don't
assume
that
they'll
get
get
ready
in
time
like
actually
and
do
the
manual
step
of
cleaning
up.
A
One
of
the
other
learnings-
this
was
a
fun
one
too
that
we
had
and
I
want
to
share
this
one,
because
I
think
this
one
is
interesting
for
anyone
who's
sort
of
developing
on
Koreans.
You
don't
have
to
be
operating
kubernetes
for
for
this
case,
so
I
am
for
service
accounts.
It's
a
feature
we
recently
launched
its
using
kubernetes
projected
protected
service
account
volumes
and
projected
projected
token
volumes.
It's
a
way
that
you
can
it's
a
feature
we
recently
launched
that
lets.
Pods
get
AWS
identities.
A
So
if
you
want
a
pod
to
assume
an
AWS
role
to
say
to
access,
DynamoDB
or
s3,
or
something
else
like
that,
this
is
the
feature
for
that
and
you
can
have
different
roles
for
different
pods.
Even
on
the
same
node,
you
don't
have
to
have
all
of
your
pods.
You
use
the
same
AWS
role
or
using
actual
key
pairs.
A
So
the
way
this
I
won't
get
into
the
the
entire
feature,
but
the
way
that
you,
as
a
user
of
kubernetes,
consume
this
there's
a
little
bit
of
setup
involved,
but
you
annotate
your
service
account
you
put
an
annotation.
That's
got
this
eks
Amazon
AWS,
ro
Laren,
and
you
put
your
your
role,
aren't
as
a
eval,
you
that
you
want
pods
that
are
using
this
service
account
to
assume
as
we.
A
This
is
sort
of
a
convenience
for
you.
So
when
you
do
this,
we
have
a
little
bit
of
magic
in
the
backend.
The
magic
is
again
getting
in
which
admission
webhook
no
surprise
right.
So
when
you
submit
a
pod
that
uses
that
service
account
that
you've
annotated,
we
have
a
web
hook
that
ingests
your
pod
and
then
based
on
that
annotation
will
update
the
pods
back
so
will
do,
will
add
sort
of
all
the
kubernetes
config
that
that
you
could
add
yourself
and
wouldn't
the
annotation
and
the
this
web
hook.
A
For
but
again
it's
a
convenience
thing.
So
all
you
can
see
the
sort
of
the
line
there
and
everything
below
the
line
gets
added
by
the
web
hook.
Everything
above
the
line
is
what
you
define
and
if
again,
if
you
defined
existing
volumes
and
environment
variables,
that
would
all
work
this
gets
appended
at
the
end.
A
So
what
sort
of
what's
the
data
path
here?
Well,
it
looks
very
similar
to
that
that
first
bug
that
we
talked
about
when
I
as
a
user
go
to
create
a
pod
in
the
base
case.
I
expect
it
to
just
go
and
create
right,
like
happens
normally
pretty
fast
time,
if
you
have
an
ec2
instance
ready
and
it's
got
available,
CPU
and
memory
or
other
resources
that
are
on
your
your
pops
back
like
that.
A
That
happens
pretty
fast,
so
just
for
for
measurements
that
we
took
on
this
the
base
case
of
creating
say
five
thousand
pods
across
the
five
hundred
nodes
p100.
So
timed
till
every
one
of
those
500
pods
was
actually
created
around
ballpark
five
five
minutes
14
seconds.
So
that's
actually
that
yeah,
that's
pretty
good,
like
if
you're
actually
doing
a
like
zero
to
two
that
roll
out
that
that's
pretty
fast
as
we
added
this
webhook
before
we
launched
again.
A
This
wasn't
a
fortunately
not
a
customer
impacting
event,
but
before
we
launch
this,
as
we
tried
this
out,
those
same
numbers,
five
hundred
five
hundred
nodes,
five
thousand
pods
as
you
the
user,
create
that
in
the
API
server
what
was
happening
is
the
API
server
reaches
out
to
this
web
hook,
hosted
on
the
control
plane.
The
web
hook
was
actually
calling
back
to
describe
to
say:
okay,
hey
I,
see
the
service
account
on
the
pod.
A
Does
that
service
account
have
the
annotation
or
not,
and
then
getting
that
back
response
back
from
the
API
server
and
then
responding
to
the
API
servers
request
and
saying:
hey:
here's,
your
mutated
pod
or
you
know
it
doesn't
happen.
There's
no
annotation!
Here's
just
the
pod!
Keep
it
going
because
of
all
this
back-and-forth.
It
seems
like
it's
just
a
serve
account
right,
like
you're.
Looking
up
just
a
string
on
a
service
account
that
shouldn't
be
that
that
long,
the
to
create
all
those
what
5,000
pods
the
the
p99
was
actually
46
minutes
or
PP
100.
A
So
that's
like
what
not
nine
hundred
and
almost
a
thousand
percent
worse
right,
like
that,
that's
unacceptable!
You
wouldn't,
like
all
your,
even
if
your
pods
weren't
weren't,
using
this
Web
book
at
all,
like
that,
was
still
the
case.
It
was
that
slow
and
also
the
other,
like
worst
part
of
this,
was
only
just
over
half
of
the
pods
were
actually
getting
correctly
mutated.
That
needed
to
be
so
when
you're
writing
a
controller,
you
want
it
to
not
be
slow
right.
So
what
was
our
fix
for
this?
A
Well,
we
actually
ended
up
adding
a
cache
so
an
our
web
hook
as
it
starts
up
it
had
this.
It
basically
requests
it.
What
are
all
the
surface
accounts
in
the
cluster
and
then
keeping
a
map
just
in
memory,
because
it's
just
strings
of
surface
accounts
to
annotations
and
then
keeping
an
open
watch
for
updates
deletions
all
that
kinds
of
stuff.
So
we
could
just
update
our
local
cache
whenever
that
happens.
A
So
now,
when,
whenever
you
as
a
user,
go
to
create
a
pod
or
the
the
replica
set
or
Dana
site
controller,
whatever
creates
your
pod,
it's
just
a
single
back
and
forth,
and
it
creates
your
pod.
And
so
after
adding
this
cache
back
there
to
the
web
hook,
RP
90
or
P
100
was
back
down
to
slightly,
maybe
one
second,
more.
It's
almost
indistinguishable
for
that
same
same
same.
B
A
Of
number
of
nodes
and
pods
so
kind
of
the
lessons
learned
here
when
you're
writing
like
interacting
with
kubernetes,
especially
when
you're
writing
like
a
some
sort
of
mutating
or
validating
admission
web
hook
like
deep
web
hooks
a
stateless
as
possible,
and
when
you,
when
you
can't
do
that,
like
you
need
some
other
data
to
reason
about,
do
I
make
changes
to
this
web
hook
or
not
like
it's
typically
a
good
idea
to
add
a
cache.
If
you
use
open
OPA
open
policy
agent,
they
actually
it
does
this
it
has.
A
It
doesn't
actually
request
from
the
API
server
for
every
resource
that
you're
putting
policy
on
it
actually
has
its
own
sort
of
sidecar
cache
to
check
hey.
Is
this?
What's
the
state
of
this
object,
or
this
other
object
that
I
need
a
reference
and
then
also
one
of
the
things
is
that
was
another
lesson
learned
was
always
be
measuring
your
changes
these
these,
these
things
are
sort
of
subtle
and
like,
if
you're,
not
not
paying
attention
like,
especially
if
you're
not
operating
on
large
scales
and
you're
like
oh,
it
just
works
on
this
small.
A
You
know
three
nodes,
three
pods
or
whatever,
like
you,
might
not
even
notice
the
difference,
but
but
keep
measurements
of
what
you're
doing
mm
I
actually
think
we're
out
of
time
for
questions,
but
I
want
to
say
thank
you
all
for
coming.
I
work
with
a
lot
of
awesome
people
who
make
eks
awesome
and
make
it
happen
and
I'll
be
available
all
around
at
the
end
for
questions,
but
I
think
we
could
take
like
two
or
three
questions:
Jeffrey
yeah,
all
right.