►
From YouTube: Dr etcd; or; How I Learned to Stop Worrying and Love the Datastore - Nick Young, VMware
Description
Join us for Kubernetes Forums Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Dr etcd; or; How I Learned to Stop Worrying and Love the Datastore - Nick Young, VMware
If you’ve deployed your own Kubernetes cluster, care and feeding of an etcd cluster is a necessary evil. At the end of the day, etcd is the place where your cluster’s buck stops, and despite the marketing hype, operating an etcd cluster at scale is not a set and forget experience.
This talk tells a story of how my team grew from etcd novices to delivering a well-monitored, reasonably resilient, etcd system that could be upgraded in less than half an hour per cluster, online, with no downtime.
After this talk, you will have:
- a better understanding of etcd's sharp edges, and what you can do to avoid catching yourself on them
- some insights on key etcd metrics to keep an eye on and why
- what we tried with upgrades, what worked, and what didn't, and how you can avoid stepping in our potholes
- some war stories, like when we accidentally made 80k namespaces in Kube and filled our etcd.
A
Hey
there,
everyone-
the
you
are
now
here
to
see
me
talk
about
dr.
NC
d
or
how
I
learned
to
stop
worrying
and
love
the
data
store.
As
you
can
tell
it's
a
dr.
Strangelove
reference
and
importantly,
doctorate
CD
is
not
me
just
like
in
the
movie
that
character
is
barely
in
this
talk.
That
is
literally
it
the
joke
is
the
joke,
is
over
the
joke
is
done.
Please
don't
I,
don't
want
anyone
to
think
that
I'm
trying
to
claim
that
mantle
for
myself,
okay,
okay,
so
but
what
I
mean?
A
What
am
I
going
to
talk
about?
Well,
I
kind
of
been
dissipated
that
you
probably
here
for
this
lots
of
ATD
war
stories,
you're
actually
going
to
get
some
ATD
war
stories,
but
there'll
be
a
couple
of
I
want
to
kind
of
give
you
some
tips
about
upgrades
and
how
managing
database
size
and
speed
is
super
important
when
you're
managing
eightydaze.
A
So
let's,
let's
go
over
the
basics,
just
in
case
anyone's
wandered
into
the
wrong
session.
So
a
TD
is
a
distributed,
reliable,
key
value
store
for
the
most
critical
data
of
a
distributed
system.
That's
straight
from
the
website!
You
know,
but
the
important
parts
for
this
talk
and
pretty
much
every
one
of
these
later
points
is
going
to
be
important.
Later
is
users
rafts
for
consensus,
which
means
that
there's
an
elected
leader,
that's
the
that
holds
the
leader
lock.
A
It
is
based
on
B
bolt,
which
stores
the
whole
database
in
memory
with
occasional
flushes
the
disk
and
it
stores
it.
It
actually
acts
like
a
git
log
a
little
bit.
It
stores
all
transactions
on
all
keys
unless
you
tell
it
otherwise,
you
know
it's
boiler
for
later.
Cooney's
does
tell
it
otherwise.
So
that's
why
you've
never
noticed
this.
You
are
probably
using
it
for
kubernetes
and,
lastly,
and
most
importantly,
if
you
have
a
problem
with
your
cluster,
you
have
a
problem
with
your
cube
cluster
and
you
have
had
probably
having
a
bad
day.
A
Alright,
maybe
you
don't
know
it
yet,
though,
so
when
you
know,
when
I,
when
I
in
the
team
I
was
working
with
first
started
using
a
TD,
there
was
one
thing
that
we
were
most
worried
about.
Upgrades
upgrades
going
wrong,
and
so
we
spent
a
lot
of
time
and
effort.
Thinking
about
you
know:
how
do
a
TD
upgrades
work?
How
does
the
you?
What
are
the
failure
modes
for
today
upgrades?
How
can
you
make
this
whole
thing
easier
to
manage
and
easier
to
actually
administer,
so
we
set
some
ground
rules
at
the
time.
A
Firstly,
we
wanted
to
have
separate
sed
notes
from
the
control
from
the
rest
of
the
control
plane,
because
we
were
really
worried
about
blast
radius,
and
you
know
if
something
goes
wrong
with
one
of
them.
We
didn't
want
it
to
affect
the
other
one
at
CDs,
your
store
of
truth.
It's
your
truth,
repository
for
for
your
kubernetes
cluster,
and
so
you
really
want
to
make
sure
that
you
know
that
something
going
wrong
on
your
API
servers
and
control
plane
doesn't
mess
with
your
thing.
I
think
this
is
a
slightly
old-fashioned
rule.
A
Now
most
people
seem
to
be
completely
cool
with
running
their
stuff.
On
the
on
the
send
control,
plane,
yeah
I
mean
personally
I'd
still
be
a
little
bit
iffy
about
it,
but
it's
definitely
a
much
lesser
thing
than
it.
Then
it
used
to
be.
We
wanted
to
make
the
nodes
immutable
as
much
as
we
could.
Everyone
in
the
team
that
I
was
in
at
the
time
is,
was
you
know
old
school
sis
admins?
We
had
done
a
lot
of
sis
admin
stuff.
A
We
had
found
that
immutable
infrastructure
saves
you
from
so
many
problems
that
we
really
wanted
to
try
to
do
that
as
much
as
good.
Also,
we
were
running
in
AWS,
and
so
we
wanted
to
try
and
use
whatever
AWS
primitives.
We
could
to
make
the
to
make
things
easier
to
make
AWS
work
for
us.
So
we
tried
a
few
things.
We
started
out
with
static
named
nodes
that
were
all
managed
in
terraform,
and
so
you
know
you
provision
the
nodes
in
terraform
you
provision
the
volume
associated
with
the
nodes
in
terraform.
A
You
know
pretty
naive
implementation
kind
of
worked
as
long
as
you
didn't
need
to
do
an
upgrade.
We
found
a
few
sharp
edges
pretty
quickly
with
the
upgrades
that
I'll
go
into
a
bit
later,
but
so
we
tried
another
thing
which
was
trying
to
make
it
so
that
the
NC
Dino's
were
in
an
SG.
That
would
be
that
would
order
rotate
whenever
you
change
the
config.
A
We
did
find
something
good,
but
later
here
we
are
and
here's
later
the
first
thing
was
we
used.
We
stood.
We
ended
up
using
an
immutable
OS
petition,
so
ACD
machines
are
Kouros
machines.
Within
with
this
immutable
OS
petition
by
immutable,
we
don't
mean
nothing
ever
changes
on
it.
We
mean
we
don't
care.
What's
on
it,
that's
because
we
have
a
separate
data
partition
which
is
mounted
by
a
small
program
at
boots.
So
you
know
WS.
Normally
you
provision
in
easier
to
instance,
but
over
birds
or
good.
A
You
provision
a
volume
as
well
that
gets
Auto
attached
and
whatever
both
ec2
and
or
whatever
tool,
you're
using
manage
that
like
make
the
lifecycle,
concurrent
and
so
the,
but
what
we
did
is
just
make
it
so
that
that
was
not
the
case.
There
is
a
volume
that's
assigned
to
this
to
each
CD
instance
and
is
an
at
boot.
A
There's
a
little
go
service
that
runs
that
picks
the
ATD
instance
that
you've
been
told
to
and
attaches
it
to
the
to
the
instance
and
that
what
that
does
is
that
when
you,
when
you're
doing
terraform,
alright,
I'm
trying
to
use
it
this
one.
Why?
While
you're
trying
to
you
when
you're
mounting
this
petition,
you
can
make
sure
that
that
the
petitions
life
cycle
is
100%
separated,
so
there's
no
way
that
that
data
can
get
deleted.
A
When
you
terminate
the
instance
again,
that's
a
that
will
be
important
later
thing,
but,
and
the
last
thing
that
was
important
was
that
we
put
life
cycle
protections
on
all
the
stuff
for
the
SED.
So
even
if
you,
if
someone
changed
the
change,
something
about
their
TD,
terraform
will
never
ever
go
and
actually
delete
that
delete.
That
instance,
whether
a
person
has
to
actually
do
the
deletion.
A
It's
a
bit
sucky
it,
but
in
a
mint
that
there
is
a
manual
process,
but
on
the
balance
of
it
we
ended
up
deciding
that
having
a
manual
process
bought
the
place
that
stores
the
sent
the
source
of
truth
for
your
cluster
was
probably
okay
again.
That
was
because
our
numbers
were
less
than
a
hundred,
so
it's
doable.
If
you're
running
thousands
of
clusters,
then
I
feel
for
you,
but
and
also
you
know,
you
probably
should
not
be
running
managing
your
tea
days.
A
Unless
you
work
for
AWS
I
guess
so
what
I
do
differently
don't
manage?
Etsy.
Do
yourself
use
a
managed
service
that
doesn't,
for
you
make
it
someone
else's
problem?
If
you
are
managing
your
own
Ltd,
you
want
to
upgrade
to
3.4
as
soon
as
it's
stable
and
there's
a
new
node
state
called
learner.
That
basically
means
when
you're
upgrading,
instead
of
joining
the
cluster
and
being
part
of
the
consensus
before
it
has
populated
all
of
the
data
from
the
rest
of
the
cluster
it
can.
A
It
learns
first
before
it
becomes
part
of
the
election,
so
it's
impossible
for
the
alert
node,
that's
in
learner
mode
to
cause
like
a
split
brain
via
lots
of
quorum.
So
this
is
really
good.
It
makes
the
it
makes
the
upgrades
heaps
safer.
It's
introduced
right
now.
The
it's
intrument
for
is
invader.
Now,
I
think
it's
only
just
introduced
it's
not
by
default.
In
three
point:
five.
A
So
here's
the
real
meat
of
why
you
hear
all
the
stories
about
all
the
times.
I
didn't
go
well,
so
the
first,
the
main
is
sort
of
two
main
classes
of
problems
that
we
had.
The
first
one
was
leader
election
storms.
Now,
leader
election
storms
are
when
you
dare
there's
just
continuous
leader
election,
and
so
nothing
can
actually
happen
in
the
class.
Stop
because
there
must
be
a
leader
elected
for
the
cost
of
to
be
able
to
write
until
that.
The
second
thing
that
happened
was
that
we
filled
the
database.
A
A
CDs
database
by
default
is
2.1
gigabytes.
If
you
put
2.1
gigabytes
worth
of
data
in
it,
you
will
have
a
bad
day.
We
did
that
by
breaking
the
compaction,
which
is
what
keeps
the
database
small
and
sometimes
by
just
putting
too
much
stuff
in
there.
So,
firstly,
let's
talk
about
leader
election
storm,
some,
so
basically
it's
continual
raft
elections
mean
that
no
rights
can
happen.
A
This
there's
a
few
ways
that
it
can
happen,
but
they
all
boil
down
to
the
same
root
cause
which
is
timeout
expiry,
the
there's
a
heartbeat
timeout
and
an
election
timeout.
The
heartbeat
is
basically
all
the
nodes
need
to
talk
to
each
other,
and
if
they
haven't
heard
from
leader
in
a
certain
amount
of
time,
then
they're
like
hey
I,
don't
have
a
letter
anymore.
A
We
better
call
an
election
and
there's
also
an
election
timeout
where,
if
you
call
an
election
and
there's
no
election
result
after
a
certain
amount
of
time,
then
you'll
call
another
election.
Importantly
here
writing
to
the
database
is
a
blocking
operation
and
can't
happen
without
the
leader
lock.
So
if
there's
no
leader,
there's
no
writing-
and
you
know,
all
of
your
latency
is
very
very
high
because
of
the
we
found
were
basically
anything
that
adds
enough.
Latency
to
your
rights
can
cause
this
disk
speed.
A
If
you
have
slow
disks,
because
a
CD
stores
everything
in
data
but
in
memory
and
saves
things
out
to
disk
periodically,
if
that
saving
process
out
to
disk
periodically
takes
too
long,
it
is
also
a
blocking
operation
and
it
can
make
if
it
takes
longer
than
your
heartbeat
timeout.
Then
your
node
will
not
respond
and
your
and
you
will
be
an
election,
and
if
that
happens
on
all
the
nodes,
all
the
time,
then
you
are
have
an
election
storm
and
the
same
thing
can
happen
with
network
latency
a
bit
harder
to
have.
A
It
happen
with
network
latency
if
you're
running
in
one
of
the
big
clouds,
but
you
can't
happen
so
what
to
do
about
it?
Firstly,
I
know
this
sounds.
This
is
the
thing
that
everybody
says
and
you're
always
like
who
don't
be
the
guy
who
just
says
read
the
docs,
but
in
this
case
the
tuning
MD
documentation
inside
the
SED
repo
is
a
complete
goldmine
for
all
of
these
things.
It
has
exactly
what
each
setting
does
exactly,
how
it
works
and
some
tips
on
how
you
should
tune
it
as
well.
A
You
know
if
it
says
like
if
you
are
doing
this,
you
should
increase
this
value.
If
you're
doing
this
decrease
this
value,
and
so
on
so
yeah
read
that
and
then
tune
your
timeouts
to
make
sure
that
it
meets
the
sort
of
requirements
and
the
performance
characteristics
of
your
load
watch
the
disk
metrics,
particularly
how
long
it's
taking
to
F
sync,
when
you're
actually
doing
the
flush
out
the
disk
and,
lastly,
watch
the
leader
elections
total
metric.
A
If
that
is
going
up
a
lot,
then
you
have
this
problem,
but
it
hasn't
been
visible
to
you
yet
so
you
you
can
have
this
problem
and
it
can
just
impact
latency
enough
to
to
be
causing
some
elections,
but
you
may
not
notice
it
if
there's
not
a
lot
of
load
in
your
cluster,
so
the
other
problem
was
filling
in
the
database.
It
is
surprisingly
easy
to
do.
2.1
gigabytes
feels
like
a
lot
data,
but
yeah.
It's
pretty
easy
for
a
runaway
cue
process
to
generate
that
much
data.
A
So
the
one
of
the
things
that
we
did
was
the
you
by
default.
A
TD
will
keep
all
revisions
of
all
transactions
on
all
keys
right.
So
there's
a
whole
lot
of
dimensions
there.
It
is
very
easy
to
have
that
the
database
fill
up
if
you
let
it
in
this
way,
but
by
default.
Kubernetes
will
compact
for
you.
That
does
require
root
privileges.
So
what
the
first
time
we
had
this
problem
was
that
we
were
like
are
having
that
API
serve.
I
have
to
have
root
privileges
across
the
network's,
not
ideal.
A
Let's
have
a
you
know,
slightly
less
privileged
role
so
that
the
C
C
we
can
control
access
to.
It
only
have
root
privileges
from
the
box.
Great
no
problems.
Let's
do
that.
We
do
that
and
then
you
sort
of
24
to
48
hours
later
we
had
a
bad
day
because
the
database
filled
up
and
that's
because
only
a
user
named
root,
or
this
was
at
the
time
only
a
user
named
root
has
the
permissions
to
perform
the
compaction
operation.
A
You
can
try
it
and
it
will
just
not
do
anything,
and
so
here,
yes,
and
so
if
the
database
ever
gets,
if
it
actually
gets
to
maximum
size,
an
alarm
gets
tripped
the
database
flips
into
read-only
mode,
and
you
will
flip
into
sad
mode,
the
you
and
but
you
can't
it
actually
requires
human
intervention
to
unflush.
If
you
manage
to
do
things,
there's
a
there's,
a
command,
you
have
to
run
to
say
no,
it's
definitely
ok
to
write
stuff
back
to
the
database.
A
Now,
like
I,
said
we
managed
to
by
getting
the
compaction
settings
wrong
and
also
we
had
a
couple
of
runaway
things.
One
time
yeah
this
one
time
at
work.
We
ended
up
with
80,000
namespaces
in
our
cluster.
Ask
me
about
that.
Another
time.
It's
a
slightly
long
story,
so
feasting
the
size
exceeded.
Errors
is
actually
surprisingly
tricky
because
your
database
is
screwed
in
is
in
a
position
where
it
can't
make
any
rights
that
includes
rights
that
it
would
need
to
do
to
reduce
the
size
of
the
database.
Lucky
you.
A
So
what
you're
going
to
do
is
stop
all
the
load
on
each
machine,
compact
and
defragment.
The
database-
this
is
a
manual
operation
if
someone's
gonna
have
to
go
on
and
type
the
right
commands
for
once.
You've
done
that
you
need
to
allow
the
API
servers
to
talk
to
the
database,
but
you
need
to
stop
anyone
else
talking
to
the
API
servers
so
that
the
reconciliation
loop
can
catch
up.
A
A
The
important
thing
here,
though,
is
that
an
LCD
outage
is
a
control
plan
outage
only
so,
while
you
were
doing
this,
yes,
you
are
having
a
bad
day.
Nothing
is
great
for
you,
but
all
the
people
running
workload
on
your
cluster.
They
probably
won't
notice
unless
they
are
trying
to
do
really
dynamic
things
with
the
API.
If
they're
just
running
services
as
I've
seen
they're
processing
web
requests,
they
probably
won't
even
notice,
but
me
so
it
is
a
C.
It
was
not
a
workload
outage.
A
You
know
failing
over
to
another
class
that
use
a
good
idea.
If
you
can
yeah.
Hopefully,
you
have
failover,
if
you
don't
good
luck
to
you,
but
and
again
in
the
in
a
TD
repo
there's,
a
maintenance
got
MD
that
has
really
great
examples
of
how
to
how
it
can
help.
You
fix
this
the
run
book
that
one
of
the
guys
ended
up.
A
Writing
was
literally
you
know,
sort
of
take
took
big
cues
from
this
thing,
so
how
you
can
not
be
a
cowboy
man
here,
let's
think
about
all
the
things
you
can
do
to
try
and
avoid
having
a
sad
day
so
avoid
adding
and
removing
nodes
for
upgrades.
If
you
can,
it's
super
easy
to
accidentally
lose
quorum
when
you
lose
quorum,
there
are
no
longer
any
guarantees
that
the
data
that's
in
your
cluster
is
correct.
A
Most
of
the
time,
maybe
you
might
be
ok,
but
you
don't
really
know
for
sure,
and
so,
when
you're
upgrading
the
normal
processes,
you
add
a
node
on
the
new
version,
then
you
remove
a
node
of
the
old
version.
When
you
add
a
node
of
the
new
version,
you
change
the
quorum
to
be
two
nodes,
and
so,
when
you
remove
the
node
of
the
old
version
until
you
have
finished
removing
that
node,
you
are
already
at
n
minus
1.
A
A
So
what
the
important
things
here
were
that
an
NC
Dino's
identity
comes
from
three
things:
the
node
name,
the
DNS
name,
that
it
refers
to
itself
and
cluster
members
of
by
the
IP
address
and
the
data
that
it
has
in
a
started
directory.
If
you
have
all
three
of
those
things
as
far
as
a
TD
is
concerned,
that
is
the
same
machine.
Even
if
it's
not
actually
the
same
machine,
so
you
can
swap
the
so.
A
What
you
can
do
is
shut
down
a
machine,
the
in
our
case
we
would
throw
away
the
entire
OS
petition
and
then
start
up
a
new
machine
with
the
same
name.
The
same
IP
address
a
different
OS
and
different
HDD
version,
but
the
same
data
data
directory,
because
those
three
things
were
the
same
as
far
as
ETD
was
concerned.
This
is
the
same
member.
A
It
just
happens
to
be
running
a
new
version
now
everything's
everything
sweet
and
the
other
thing
that
is
really
nice
about
it
is
that,
once
you
hit
all
all
the
nodes
in
your
cluster
running
the
new
version,
then
the
cluster
will
self
upgrade
it
will.
It
runs
at
the
lowest
possible
version
amongst
all
of
the
members.
So
it's
only
once
you
have
all
the
members
upgraded
to
the
new
version
that
the
functionality
of
the
cluster
will
actually
be
upgraded.
A
So
if
you
can't,
if
you
must
add
or
add
or
remove,
add
and
remove
nodes,
really
really,
you
need
to
understand
exactly
how
the
quorum
works
and
the
failure
cases
that
could
happen
and
have
pretty
good
processes
for
how
you
make
sure
you
do
this
quickly
and
reliably
and
checks
along
the
way,
all
that
sort
of
so
for
watching
the
database.
There's
some
metrics,
you
should
watch
here's
the
exact
names
in
case
you
wanna,
take
a
photo.
The
the
each
one
of
these
is
pretty
important.
The
first
one
is
the
the
log
F
sync
Eurasian.
A
The
second
one
is
how
long
it
actually
takes
to
commit
the
full
flush
to
disk.
The
third
one
is
how
many
leader
changes
have
been
seen
by
the
by
this
node
that
one
is
yeah,
that's
the
sort
of
the
canary
metric
that
says
hey
something
might
be
wrong
and
the
last
one
is
the.
If
you
monitor
nothing
else
from
your
at
CD.
This
is
the
thing
you
gotta
monitor.
This
is
the
size
in
bytes
of
the
data
base.
A
If
that
number
hits
2.1
gigabytes
your
data
base
flips
into
read-only
and
again
you
flip
into
said,
maintenance
and
tuning.
Are
your
friends.
You
know
read
those
things
love
them.
They
are
really
surprisingly
good
documentation
for
something
that
sits
in
an
MD
inside
of
get
up
early
and
then
the
usual.
The
usual
sort
of
operational,
suspect,
stuff
right,
a
recovery
run
book
and
try
it
out,
and
you
make
sure
you
try
it
out.
Don't
just
write
the
run
book
once
have
someone
you,
and
this
is
where
wargames
can
help.
A
It's
actually
really
fun
to
you,
know:
break
someone
else's
dead,
CD
cluster
and
be
like
see.
If
you
can
figure
out
what
I
did
you
know?
Maybe
you
sat
in
there
and
added
like
40,000
services
and
now
there's
things
out
of
space
and
yo
and
then
they're
like
hard.
Now
you
try
and
fix
it,
and
then
they
have.
They
can
take
the
run
book
and
try
and
actually
see
if
it
works.