►
Description
Want to view more sessions and keep the conversations going? Join us for KubeCon + CloudNativeCon North America in Seattle, December 11 - 13, 2018 (http://bit.ly/KCCNCNA18) or in Shanghai, November 14-15 (http://bit.ly/kccncchina18).
Understanding Distributed Consensus in etcd and Kubernetes - Laura Frank, Codeship (Intermediate Skill Level)
The Raft consensus algorithm is at the core of etcd, allowing you to run highly available services across your Kubernetes cluster. But how exactly does quorum work, what happens when a node goes down, and how do all of the machines agree on the actual state of your services? Where is the Raft log stored, and how can you inspect it? Come away with deeper understanding of two of Raft’s main responsibilities, log replication and leader election, and be better equipped to make informed choices about your service declarations and cluster configurations.
About Laura
As the Director of Engineering at CloudBees and a Docker Captain, Laura's primary focus is making tools for other developers. At CloudBees, she works on improving the Docker infrastructure of the Codeship product and overall experience for all users of the CI/CD platform. Previously, she worked on several open source projects to support Docker in the early stages of the project, including Panamax and ImageLayers. She currently lives in Vienna.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
All
right,
hello,
everybody,
I'm
gonna,
start
30
seconds
early,
because
I
have
the
microphone
and
I
can
so
I'm
gonna.
My
name
is
Laura
Frank
I
am
the
director
of
engineering
at
cloudBees,
formerly
coach
ship.
We
just
got
acquired
about
three
months
ago,
so
we're
very
happy
to
be
joining
up
with
a
bunch
of
other
people
trying
to
solve
DevOps
problems.
A
However,
I'm
not
here
to
talk
about
CI
or
CV,
or
anything
like
that
I'm
here,
to
talk
about
distributed
consensus,
which
is
a
really
scary
thing,
but
I'm
gonna
try
to
make
it
a
little
bit
less
scary
for
you,
I'm
gonna
go
over
a
couple
main
topics.
Today.
First
is
demystifying.
This
very
intimidating
phrase
called
distributed.
Consensus.
We're
gonna
talk
specifically
about
the
raft
consensus
algorithm,
what
it
is
where
you
find
it,
why
it's
there?
A
We
will
talk
about
quorum,
which
is
essential
to
understand
and
if
you're
operating
any
system
that
depends
on
distributed,
consensus,
we'll
talk
about
leader,
election
and
log
replication
and
then
just
maybe
at
the
very
end,
I
will
break
a
cluster
because
it's
my
favorite
thing
to
do
so:
I'm,
not
necessarily
showing
best
practices.
If
you're
operating
a
cluster
in
production,
that's
not
really
the
point
of
this
talk
I'm
here,
to
give
you
a
little
bit
more
theoretical
background
for
the
sort
of
theoretical
underpinnings
of
what
makes
the
kubernetes
world
go
round.
A
So,
let's
get
started
first,
with
distributed
consensus
and
I,
really
like
saying
this
term
or
like
bringing
it
up
at
a
party
I'm
real
fun
at
parties.
It
sounds
really
fancy,
but
what
it
means
is
just
getting
more
than
one
party
to
agree
on
something
and
that
those
parties
are
not
all
located
in
the
same
place.
So
we're
really
focusing
on
agreement
and
the
reason
that
we
have
to
do.
That
is
because
we've
sort
of
made
it
our
own
problem,
her
we've
sort
of
created
the
problem
for
ourselves.
A
So
we
used
to
not
really
have
to
worry
about
distributed
consensus
because
the
world
was
really
easy.
We
had
a
client
that
maybe
made
a
request
to
a
server
to
update
a
value
and
the
server
kind
of
knew
what
the
values
were
and
said.
Yep
cool,
I'm,
gonna
append
that
value,
and
then
my
new
value
is
seven.
For
example,
we
use
tools
like
kubernetes,
though,
because
we
don't
want
that
model
anymore.
We
don't
want
there
to
be
a
single
point
of
failure,
just
one
database.
A
We
want
redundancy
and
fault
tolerance,
and
that
means
having
multiple
entities
in
a
lot
of
cases.
That
means
having
a
management
group
or
consensus
group
that
is
comprised
of
more
than
one
machine.
So
we
have
to
get
all
of
those
machines
to
agree
on
the
state
before
we
can
take
an
action
on
our
cluster.
A
So
in
this
distributed
model,
here's
a
visualization,
so
we
have
a
client
making
a
request,
but
then
that
request
needs
to
be
decided
upon
by
a
manager
group
and
in
a
manager
group
there's
always
a
leader,
that's
the
the
node,
that's
taking
in
the
requests,
and
then
there
are
followers
who
are
participating
in
that
consensus
group,
but
aren't
necessarily
in
control
of
making
making
the
final
decisions.
So
they'll
get
requests
to
say
hey
from
the
from
the
leader
like
hey.
Can
this
happen
is
this?
A
A
A
This
has
been
I,
think
one
of
those
underground
parts
of
our
system
that
we
didn't
really
talk
about
much,
but
in
the
last
couple
of
years
it's
gotten
a
lot
more
popular
and
a
lot
more
visibility,
which
makes
me
happy
because
I
find
it
very
interesting
raft
is
an
algorithm
to
manage
consensus
based
systems
such
as
container
orchestrators.
It
can
be
certainly
used
in
other
kinds
of
applications
or
other
other
ways
as
well.
A
I
would
also
encourage
you
not
to
get
too
intimidated,
maybe
by
the
word
algorithm
in
raft,
because
it
is
a
complex
system
and
it's
just
a
system
of
patterns
and
I
think
it
can
be
really
accessible.
It
just
sounds
a
little
bit
intimidating,
which
hopefully
you'll
you'll
find
out
by
the
end
of
this
talk
that
it's
easier
to
understand.
It's
designed
to
be
easier
to
understand.
So,
if
you
are
from
a
traditional
CS
background,
you
might
have
studied
paxos
or
multi
Paxos.
A
That
was
a
bad
time
in
in
my
in
my
study,
so
the
laughter
tells
me
it
was
probably
a
bad
time
for
you
as
well
raft
sort
of
came
out
as
like
the
opposite
of
that.
Like
hey,
this
was
really
painful.
Why
don't?
We
figure
out
a
way
to
make
this
not
only
perform
as
well,
but
also
a
little
bit
easier
to
understand
so
that
the
people
who
are
responsible
for
operating
systems
that
use
this
algorithm
can
actually,
you
know,
understand,
what's
going
on
and
be
able
to
kind
of
process
it
at
a
human
level?
A
A
It's
also
it's
like
really
hard
to
implement
yourself
and
for
that
reason,
raft
is
used
in
a
lot
of
places.
So
orchestration
systems
I
think
are
the
most
visible
way
or
most
visible
place
that
that
raft
is
used
so
kubernetes
using
SCD
and
then,
by
extension,
every
single
system
that
uses
@cd
in
any
way
shape
or
form
is
using
rafts.
Because
raft
is
embedded
inside
of
@cd
docker
uses
raft,
it's
embedded
inside
of
docker
itself.
Dr.
doesn't
have
a
reliance.
A
net
CD
for
for
their
orchestration.
Nomad
has
a
similar
approach.
A
Where
raft
is
kind
of
baked
in
the
internals.
If
you
are
familiar
with
zookeeper
zookeeper
uses
something
slightly
different
from
raft
called
the
zookeeper
atomic
broadcast,
which
is
sab,
which
is
fun
to
say
it's
similar
to
raft,
has
a
lot
of
the
same
responsibilities,
a
lot
of
the
same
patterns-
just
it's
not
raft,
but
luckily
the
patterns
are
very
similar.
So
if
you're
using
zookeeper
zookeeper
in
some
way,
you
can
kind
of
translate.
A
What
we'll
talk
about
today
onto
that
raft
has
three
main
responsibilities
wherever
it's
being
used,
the
first
one
and
probably
the
most
important
one
is
log
replication
raft.
Is
there
to
maintain
a
consistent
state
across
your
distributed?
Node
that's
participated,
nodes
that
are
participating
in
that
consensus
group
and
then,
by
way
of
that
consensus
group
to
your
entire
cluster,
we'll
talk
about
each
of
these
in
more
depth
as
kind
of
the
main
meat
of
the
talk
leader.
A
Safety
is
another
responsibility
of
raft
and
I'm,
not
gonna,
go
into
depth
on
this
because
it
sort
of
baked
into
both
log
replication
and
leader
election
safety
is
the
sense
that
any
action
that
is
happening
in
the
cluster
should
be
allowed
to
happen
and
raft
is
designed
to
make
actions
that
shouldn't
be
allowed
to
happen
impossible
to
occur
and
that's
sort
of
baked
into
both
log
replication
and
leader
election.
So
I
won't
go
deeply
into
safety
today,
but
that
was
one
of
the
design
goals
of
raft.
I.
A
Think
one
of
the
other
big
things
that
raft
is
responsible
for
is
being
easier
to
understand,
as
we
are
using
tools
like
kubernetes
more
in
production,
and
we
have
people
responsible
for
operating
them.
If
something
happens
with
your
management
group
are
your
consensus
group?
If
that
you
lose
form
or
something
goes
wrong,
it's
not
fun
and
I
think
it's
always
better
to
be
able
to
really
understand.
What's
going
on,
so
you
can
troubleshoot
and
that's
one
of
my
goals.
Also
for
this
talk
is
to
give
you
some
more
fundamental
knowledge.
A
So
the
first
thing
that
we'll
talk
about
in
terms
of
fundamental
RAF
knowledge
is
quorum,
and
this
is
a
buzzword-
maybe
it's
fun
to
say,
I
think
quorum
is
not
a
term
unique
to
computer
science
or
orchestration
systems
or
at
CD
or
raft
or
anything.
This
is
a
borrowed
term.
It
is
just
the
minimum
number
of
votes
needed
to
perform
an
operation.
I
think.
Usually
we
hear
this
term
in
government
depending
on
what
country
you're
from
sometimes
the
groups
of
elected
officials
need
quorum
in
order
to
conduct
a
vote,
and
this
is
the
same
idea.
A
It's
just
a
borrowed
concept
without
quorum.
Nothing
can
happen
because
the
system
is
not
allowed
to
do
work
and
I
think
we
have
a
general
idea
that
quorum
means
majority
and
it
doesn't
just
apply
to
nodes
being
online.
They
have
to
actually
agree
and
vote
for
the
same
thing
in
order
for
that
action
to
take
place,
so
majority
is
a
good
way
to
think
about
quorum.
It's
also
a
much
easier
word
to
understand
and
to
spell
and
to
pronounce,
but
it's
it's
I
guess
slightly
different
quorum
is
specifically
defined.
A
If
n
is
the
number
of
nodes
in
your
cluster
and
divided
by
two
plus
one,
so
the
most
important
thing
about
that
is
distinctly
and
precisely
more
than
50%
50%
does
not
constitute
quorum.
It
has
to
be
some
value
above
50%
and
thankfully,
because
this
is
just
an
easy
mathematical
expression,
we
can
fill
out
a
chart
and
you've
probably
seen
a
chart
like
this.
A
If
you
are
operating
at
CD
or
another
system
with
a
consensus
group,
this
is
a
pretty
common
thing
to
say:
okay,
how
many
nodes
should
I
have
in
my
consensus
group
in
that
cluster?
If
you
have
one
node
in
here,
I've
labeled
those
Raft
consensus
group
participants
managers,
that's
a
common
term
to
describe
them.
If
you
have
one
note.
Obviously,
quorum
is
one.
If
that
node
goes
away,
then
you
can't
do
any
more
work
on
your
cluster.
A
A
One
other
way
to
think
about
quorum
is
not
about
how
many
you
need
to
satisfy
quorum,
but
what
your
fault
tolerance
is
when
you're
making
these
choices.
So
if
we
go
all
the
way
back
up
to
1
our
fault,
tolerance
is
obviously
zero.
We
can't
survive
a
failure
of
a
node.
In
that
case,
the
same
is
true
for
two:
we
can't
survive
a
failure,
but
if
we
have
three
nodes
in
our
consensus
group
quorum
is
two.
A
That
means
one
can
go
down
or
maybe
there's
a
network
partition
or
something
happens
where
it's
unreachable
and
our
cluster
can
still
be
up
and
running
and
and
perform
new
work.
So
my
recommendation
is
that
if
you're
running
something
in
production,
especially
if
it's
making
money
five
is
sort
of
the
magic
number
because
then
you
can
tolerate
one
failure
and
then
have
one
node
down
for
maintenance.
So
just
my
experience
working
with
many
different
projects
in
production,
that
seems
to
be
sort
of
the
magic
number
again.
Odd
are
sorry.
A
Even
numbers
are
not
very
efficient
in
this
type
of
model
because
of
that
n
divided
by
two
plus
one
and
over
50%,
you
don't
actually
get
any
more
benefit.
If
you
have
one
manager,
just
adding
another,
isn't
going
to
give
you
more
fault,
tolerance
and
in
a
very
counterintuitive
point,
that's
always
mind-blowing
when
I
share
it
with
people
who
haven't
thought
about
it.
A
In
this
way,
having
two
managers
actually
doubles:
your
chances
of
losing
quorum
because
you
still
have
zero
fault
tolerance
and
now,
instead
of
having
one
single
point
of
failure,
you've
doubled
your
chances
of
having
a
node
failure
by
introducing
another
one.
So
stick
to
odd
numbers,
stay
away
from
even
numbers
and
you'll
be
good.
A
So
just
pay
attention
to
a
data
set
data
center
topology
when
you
do
that,
anticipate
that
one
of
those
Easy's
will
go
down
and
that
the
Status
page
won't
be
updated
for
that
cloud
provider
not
going
to
name
any
names.
But
there's
a
I
mean
it's
just
it's
math
it.
It's
follows
that
same
quorum
principle,
just
anticipate
failure,
I
think.
That's
a
really
good
strategy
in
general
for
distributed
computing,
a
lot
of
turnkey
solutions
that
might
stand
up
a
bootstrap,
a
cluster
for
you
magically
kind
of
have
this
functionality
in
it.
A
So
that's
quorum.
Quorum
is
going
to
become
a
very
important
thing
as
we
talk
about
both
leader
election
and
log
replication.
So,
leader
election
is
very
important
and
one
of
the
core
responsibilities
of
Raft,
because
without
a
leader
your
management
group
or
your
consensus
group
can't
do
any
work,
because
there's
no
one
to
delegate
responsibility
and
control
whether
or
not
a
log
can
be
appended
or
should
be
committed
into
the
log.
In
a
consensus
group,
there
are
really
three
states
of
nodes
that
participate.
One
of
them
is
the
leader.
A
The
leader
is
again
the
one
taking
in
the
requests
and
then
sending
messages
out
to
the
followers,
which
is
the
other
state,
to
make
sure
that
that
action
is
able
to
take
place
and
gets
confirmation
from
the
followers
commits
it
and
then
sends
that
message
back
to
the
followers
to
say:
okay,
yeah,
you
can
really
really
write
that
into
your
log.
Now,
we've
gotten
enough
votes,
there's
a
third
state
called
a
candidate
which
happens
in
leader
election
if
the
leader
is
unreachable
or
goes
offline.
A
For
some
reason,
if
you
execute
chaos,
monkey
scripts
in
your
cluster,
like
some
people
might
do,
then
your
your
consensus
group
doesn't
have
a
leader
and
we
need
to
go
into
leader
election.
The
first
follower
to
notice
that
there's
no
leader
will
put
itself
into
candidate
state
in
order
to
solicit
votes
from
the
rest
of
the
nodes
that
are
online
to
elect
itself
as
the
new
leader.
It
will
also
vote
for
itself,
there's
also
another
another
state
which
is
kind
of
an
unofficial
one
which
is
unreachable.
A
I
think
that's
important
to
remember
that
there
can
be
notes
that
are
formally
participating
in
the
consensus
group
that
could
be
unreachable,
so
that
could
be
the
downed
manager
or
a
Down
node
or
the
downed
leader.
That's
spawning
the
next
leader
election.
They
still
exist
even
if
they're
not
online,
so
I
want
to
talk
through
a
couple
scenarios
of
leader
election
and
to
do
that.
I
have
a
little
handy.
Visual
aid
actually
forked.
This
from.
Let
me
see
if
I
can
make
this
bigger
cool
yeah.
A
You
can
all
see
that
okay
I
this
from
the
author
of
raft
and
kind
of
tweaked
it
a
little
bit
for
this
talk
and
if
you've
ever
looked
at
the
secret
lives
of
data.
If
you
are
a
independently
curious
raft
consumer,
then
you
may
have
seen
something
like
this
before
I
want
to
talk
through
what
happens
during
a
leader
election,
we're
gonna
start
out
really
slow.
So
a
couple
things
that
pay
attention
to.
We
have
a
5
node
consensus
group
up
here,
starting
with
s1
s2
s3.
A
Those
are
the
names
of
the
nodes
in
the
in
the
consensus
group.
We
have
this
little
ticker.
That's
happened
going
around
and
that's
a
timeout
and
I
said
before,
like
the
first
manager
to
notice
or
the
first
node
to
notice
that
there's
no
leader
is
gonna,
put
itself
up
for
election
and
that's
exactly
what
happened
right
here
with
us
to
notice
that
there
was
no
leader,
so
it
timed
out
and
put
itself
up
for
election
solicited
votes
and
I
can
rewind
time
a
little
bit
here.
I'm
sorry
I.
A
Got
I
just
got
a
new
Mac,
a
new
MacBook
with
the
touch
bar
and
like
the
huge,
attract
pad
and
I'm
very
unhappy
with
it.
So
please
bear
with
me
there
we
go
okay,
so
we
see
that
s2
timed
out,
and
now
we
have
a
new
election
cycle
so
election
term
two,
we
see
five
spaces
for
votes
in
there,
those
little
circles.
We
can
see
that
as
two
voted
for
itself.
It
gets
a
black
circle
from
itself.
A
It's
asking
for
votes
from
the
rest
and
everyone's
online
and
everyone
votes,
and
that
means
yep
s2
gets
elected
as
the
leader,
and
we
see
that
in
red
a
little
bit
probably
hard
to
see
for
those
of
you
in
the
back.
So
that's
great
and
things
will
just
carry
on
until
oh,
no
s2
went
down
whether
or
not
it.
A
Actually,
there
was
a
hardware
failure
or
there's
a
network
partition
or
who
knows
what
could
happen,
but
the
point
is
that
it
went
down
and
just
as
in
the
last
time,
the
first
node
to
notice
that
there's
no
leader
is
going
to
put
itself
up
for
candidacy.
Now
quorum
becomes
important
here
because
we
have
one
node
that's
down,
but
since
we
have
a
five
node
cluster
quorum
is
three,
which
means
that
four
is
fine
and
we
can
keep
going
and
we
can
tolerate
that
failure.
A
A
Cool
there's
one
thing:
actually
there's
many
edge
cases
that
can
happen
in
this
and
one
that
I
find
particularly
interesting
and
one
thing:
that's
a
little
bit
different
in
raft
versus
other
implementations
like
paxos
and
multi.
Paxos
is
the
event
of
a
split
vote,
so
this
is
highly
unlikely
because
these
timeouts
are
randomized
across
all
of
the
nodes.
A
So
it's
highly
unlikely
that
two
nodes
in
your
group
have
the
same
timeout,
but
it
can
happen-
and
in
this
case
it
has
so
I
can
go
back
in
time
just
a
little
bit
and
see
that
these
two
nodes-
s
1
and
s
5
timed
out
exactly
the
same
time
and
put
themselves
up
for
candidacy
for
election
term
4
at
exactly
the
same
time.
They
also
sent
votes
out
at
exactly
the
same
time
and
they
are
voting
for
themselves.
A
A
But
what
happens
then,
because
it
looks
like
we'd
be
just
at
a
standstill
raft-
is
designed,
though,
if
there's
no
leader,
whatever
the
next
node,
is
which
in
that
case
was
s3
oops
to
notice
that
there
was
no
leader
is
gonna,
put
itself
up
for
election,
and,
let's
just
go
through
that,
so
s3
right
here
is
just
about
to
timeout.
It
doesn't
matter
the
candidacy
state
of
s
1
and
S
5.
They
don't
have
enough
votes
to
be
elected,
so
what
Raph
does
is
just
throws
out
that
cycle.
A
A
Having
a
candidate
state
of
a
node
doesn't
prevent
it
from
voting
for
another
node
as
well.
So
s
1
and
s
5
are
able
to
participate
in
that
election.
It
can
get
the
requisite
votes,
so
it
gets
four
in
this
case
and
time
marches
on
we've
just
moved
on
to
a
different
election
cycle,
so
that
is
leader
election
kind
of
a
brief
intro
to
it.
There's
a
couple
other
edge
cases
that
are
captured
by
think
that
split
vote
is
really
the
most
common
that
that
is
possible
to
happen.
A
The
next
thing
that
I
want
to
go
into
that
more
depth
on
is
log
replication,
because
it's
kind
of
similar
with
that
heartbeat
timeout
transaction
between
the
manager
and
the
and
the
followers,
but
it's
a
little
bit
more
more
complicated,
so
we're
gonna
combine
sort
of
everything.
We've
talked
about
so
far
and
talk
about
log
replication
before
we
do
that,
though,
I
want
to
make
it
clear
what
I'm
talking
about
when
I'm
talking
about
a
log
I
think
this
is
also
one
of
those
terms.
That's
like
highly
loaded,
and
it
needs
like
that.
A
Wikipedia
like
which
log
are
you
talking
about
page
when
we,
when
we
discuss
it.
This
is
the
source
of
truth
for
your
application,
I'm,
not
talking
about
log
output
or
system
log
I'm.
Talking
about.
Like
imagine
someone
hammering
values
into
a
stone
tablet,
kind
of
log,
a
record
of
truth,
so
this
is
an
append-only
time-based
record
of
data.
We
can
visualize
that
really
simply
by
imagining
just
a
list
of
you
know
some
values
that
let's
say
X
has
so
at
one
point
in
time.
A
X
had
two
and
now
we're
going
to
maybe
append
an
entry
for
whatever
the
new
value
of
X
is.
This
log
is
specifically
for
computers,
really
it's
not
for
humans.
Luckily,
for
you,
all
I
am
a
human
with
a
computer,
so
I'm
going
to
show
you
how
you
can
actually
read
the
raf
log
and
where
it's
found,
which
I
think
is
pretty
interesting
before
we
do-
that.
Let's
look
at
what
happens
in
this
log
replication
scenario
and
go
over
a
couple,
a
couple:
bad
states
with
some
data
inconsistency
and
see
how
raft
repairs
them.
A
So
in
this
case
we
have
States
for
a
log
entry
to
be
one
is
uncommitted
and
those
are
represented
by
the
little
dashed
lines.
If
we
see
s1
has
two
to
two:
those
are
all
uncommitted.
That
means
that
the
leader
has
scented
the
follower.
Please
update
this
to
two,
but
has
not
gotten
enough
votes
and
enough
confirmation
to
make
that
truth.
Yet
so
that's
uncommitted
state,
that's
sort
of
like
you
know,
pre-commit,
not
yet
in
in
the
log.
A
A
This
will
go
on
and
will
just
populate.
Whatever
nodes
are
listening
with
these
uncommitted
log
entries,
let's
move
a
little
bit
faster
and
see
what
happens
then,
when
we
do
have
quorum,
one
came
back,
which
means
that
we
can
now
start
committing
these.
So
now
that
enough
votes
are
possible,
even
though
those
nodes
knew
about
the
possibility
for
change
that
change
hadn't
yet
been
voted
on,
so
it's
not
been
committed
in
the
log
now
they
have
been
so
that's
represented
by
this
solid
line
across
or
around
the
the
log
entry.
A
The
one
really
interesting
thing
about
raft
is
what
happens
in
the
case
of
data
against
inconsistency.
So
in
this
example
we
had
know
some
nodes
offline,
but
I,
don't
know
what
they
were
doing.
They
could
have
been.
You
know,
on
a
separate
partition
and
had
a
new
election
cycle
and
had
a
new
state
of
truth.
We
don't
know,
but
now
let's
say
that
happened,
and
maybe
the
network
partition
is
restored.
What
happens
now?
We
have
two
sets
of
of
data,
and
how
do
we
repair
the
inconsistencies
and,
in
general,
the
answer
to
that
in
Raph?
A
Is
that
whatever
the
most
recent
election
term
was
that
data
gets
precedent
precedence?
So
in
this
case
we've
already
moved
on
to
election
cycle?
Three,
we
have
some
the
old
leader
which
was
s1
is
offline.
Now,
so
we
have
kind
of
like
two
conflicting
sets
of
data.
When
s1
comes
back
online,
it
still
thinks
that
it's
like
the
year
1995
or
it's
like
election
term.
Two,
it
doesn't
know
what
leader
has
been
elected.
It
will
get
in
forms
soon
by
the
new
leader.
A
A
Cool
so
there's,
of
course,
other
edge
cases,
but
I
think
that
network
partition
is
particularly
the
one
that
you
might
find
in
production
and
is
more
likely
to
happen
in
real
life.
So
if
you
are
new
to
logs
or
kind
of
new
to
log
replication
in
just
distributed
consensus,
there's
a
really
awesome
blog
post
about
logs
and
what
you
all
should
understand
about
them.
A
If
you're
gonna
be
operating
any
system
that
depends
on
the
log
I,
have
it
up
on
a
handy
little
bit,
Lee,
it's
a
LinkedIn
engineering,
blog
post,
so
highly
recommend
that
it's
a
kind
of
a
long
read,
but
a
good
one.
So
if
you
want
to
check
that
out
cool,
so
I
promise
that
I'm
a
person
with
a
computer
or
a
human
with
a
computer
which
means
that
I'm
able
to
look
at
the
logs,
so
I
actually
wanted
to
do
this
live,
but
the
Wi-Fi
was
choking
a
little
bit.
A
So,
instead
of
me,
kind
of
like
fumbling
and
waiting
for
Wi-Fi
to
error
out
I
just
prepared
this
beforehand,
but
I
can
talk
through.
Maybe
I
can
talk
through
what
this
is
so
I'm
gonna
I'm
gonna.
Do
something
like
scandalous
and
I'm?
Actually
gonna
use
docker
for
the
example
for
for
raft
instead
of
at
CD,
because
I
find
the
tooling
just
slightly
slightly
better
for
conference
talk
demo.
Specifically,
there
is
a
choice.
A
I
said
before
raft
is
embedded
directly
in
docker
and
everything
lives
in
a
directory
in
at
CD,
it's
living
in
var,
Lib
at
CD
member
there's,
a
snap
directory
and
a
wall
directory
so
snapshots,
and
then
the
right
head
log
right,
upend
log
right
ahead
log,
those
are
a
net
CD
they're
just
file,
and
it's
the
same
thing
because
raft
is
the
consistent
part
here.
So
it
doesn't
really
matter
what
tool
we're
talking
about.
That's
also
available
in
docker
and
when
I
use
this
there's
a
tool
called
raft.
A
There's
a
dump
wall
tool
in
in
swarm
kit
and
I'll
have
links
in
the
slide.
So
you
can
check
it
out
later.
It's
a
little
bit
too
much
detail
to
kind
of
try
to
process
during
a
talk,
but
it
basically
looks
at
that.
Log
crosses
it
into
something:
that's
human,
readable
and
then
spits
it
back
out.
So
I
have
a
cluster
here
of
three.
In
a
raft,
consensus
group
and
basically
what's
stored
in
the
raft
logs
is
everything
that
you
could
possibly
need
to
know
about
the
state
of
the
cluster.
A
So
we
have
the
action
that
happened.
We
know
that
index,
we
know
exactly
when
it
was
created.
When
it
was
updated,
we
can
see
the
image
we
can
see.
Resource
limits,
status
kind
of
everything
is
here
which
is
really
great,
because
if
you
have
this
raft
log,
you
can
basically
breathe
life
into
your
cluster
in
case
of
a
failure
and
there's
a
couple
different
failure
scenarios
that
I
want
to
talk
about
and
we'll
talk
about
this
raft
log
and
its
role
in
each
of
those.
A
A
You
know,
like
asterisks,
there's,
probably
some
little
things
that
you're
not
going
to
be
able
to
restore,
and
you
can't
restore
what
happened
after
you
took
the
backup,
but
you
can
kind
of
rewind
time
and
breathe
life
back
into
a
failed
cluster
if
you've
sort
of
declared
bankruptcy
and
your
incident
response
and
need
to
start
from
scratch.
As
long
as
you
have
that
backup,
you
can
do
it.
A
Generally
speaking,
wherever
raft
is
implemented,
there's
going
to
be
two
directories:
there
will
be
a
snapshot
directory
and
that
log
directory
in
wall
in
xcd
you
can
specify
where
these
come
from
or
where
they're
stored.
So
you
can
put
them
somewhere
else.
You
can
back
them
up
somewhere
else.
You
can
kind
of
define
a
disk
and
adder
of
your
choice
in
at
CD,
at
least.
So
that's
a
really
cool
feature.
A
Some
tools
will
manage
this,
for
you,
I
think
that's
becoming
like
the
new
horizon
of
kubernetes
management
is
just
to
automatically
backup
at
CD,
so
that
you
always
have
something
on
hand.
If
you,
if
your
cluster
goes
down
in
a
dumpster
fire,
you
can
just
sort
of
bring
it
back
up
if
you're
gonna
restore
from
a
backup,
so
a
net
CD.
A
You
can
do
this
the
same
thing
on
docker
again,
because
it's
not
anything
specific
to
the
tooling
it's
about
raft
and
about
that
raft.
Log
same
thing,
docker
swarm
in
it
with
that
forced
new
cluster
will
kind
of
disregard
the
cluster
configuration
you
had
before.
Take
that
log
and
start
a
brand
new
cluster
with
one
node
using
that
backup,
one
important
thing
to
keep
in
mind
and
I'm
I'm,
hoping
that
this
will
be
a
solved
problem
in
maybe
a
couple
months
time.
A
I
know
there's
a
lot
of
smart
people
working
on
this
problem
when
you
restore
from
the
backup.
Those
like
member
IPS
of
the
nodes
that
are
in
the
cluster
are
also
stored
as
part
of
that
log,
which
means
that
if
you
have
a
brand
new
cluster
at
the
IP
addresses,
are
probably
not
going
to
be
the
same
in
ED
CD.
What's
really
awesome,
and
probably
one
of
my
favorite
features,
that's
very
underappreciated
is
that
it's
super
easy
to
update
a
member
IP.
A
You
can
just
like
go
in
there
and
say:
oh
I
have
a
new
node,
that's
replacing
this
old
node.
This
is
the
new
IP
address
swarm
at
least
the
last
I
checked.
Maybe
it's
maybe
it's
different.
You
can't
really
do
that
in
an
in
a
graceful
way.
I
think
that's
one
really
great
thing
that
CD
has
for
it
if
you're
an
SRE
or
someone
that's
on
call,
that's
something
that
makes
me
sleep
a
little
bit
better
at
night.
A
So
let's
talk
about
a
soft
failure
and
I'll
demo,
a
soft
failure,
because
that's
hopefully
fingers
crossed
when
I
can
recover
from
more
easily
and
with
the
time
allotted
so
that
I
can
get
you
all
to
lunch
on
time.
So
losing
quorum
is
probably
the
most
common
failure.
It's
not
common
that
our
data
centers
may
come
up
in
smoke,
so
you
probably
want
to
practice
that
backup
recovery.
You
know
every
month
or
every
week
or
whatever
your
policies
are
practice.
A
Also
what
happens
when
you
lose
quorum
because
it
can
mean
different
things
for
your
application,
so
losing
quorum.
Just
means
that
no
new
action
can
take
place.
It
doesn't
mean
that
everything
that's
running,
falls
down
all
right
and
that's
an
important
distinction
so
understand
what
that
means
for
your
app.
If
you
have
just
a
bunch
of
services
that
are
long-running
and
you're
just
using
a
tool
like
kubernetes,
because
you
want
Phil,
you
know
fault,
tolerance
and
resistance
to
failure.
A
The
interesting
thing
about
losing
quorum
is
that
you
also
can't
add
new
nodes
to
that
management
group.
So
let's
say
you
have
three
nodes.
You
can't
just
and
two
of
them
go
down.
You
can't
just
start
a
new
node
and
add
it
to
the
group
and
try
to
like
force
quorum
to
be
regained,
because
your
management
group
can't
agree
that
it's
possible
for
a
new
manager
to
be
added.
A
So
how
do
you
survive
a
quorum
like
losing
quorum?
You
can
regain
quorum
by
bringing
the
down
nodes
back
up
like
that's
kind
of
Captain
Obvious,
probably
not
something
useful
in
the
talk,
but
it
is
there
and
I
think
it's
like.
Maybe
you
don't
think
about
that?
An
instant
response.
The
other
thing
you
can
do
is
that
this
is
sort
of
like
that
disaster.
Recovery
that
we
talked
about
before
is,
but
you
can
use
force
new
cluster
on
a
healthy
manager.
A
That's
part
of
the
management
group
or
the
the
consensus
group
and
do
new
force
new
cluster,
and
that
will
just
look
at
the
current
raft
log
that
it
already
has
there's
no
need
to
restore
from
backup
and
just
disregard
the
fact
that
there
were
at
one
point,
other
managers.
You
will
have
to
re-
add
new
new
nodes
to
that
leader
group.
If
you
do
it
this
way
and
again,
you
can't
just
add
a
new
healthy
manager
to
the
cluster,
because
that
would
require
consensus
and
that's
something
that
we
don't
have.
A
A
A
We
can
see
that
failure,
so
we
can
expect
that
that
fifth
one
will
be
rescheduled
soon.
Cool
everything's
happening
just
as
we
expect
that's,
because
we
still
have
quorum
and
things
are
fine,
but
if
I
always
makes
me
a
little
nervous,
it's
fine,
it's
just
it's
just
virtual
machines.
It's
okay,
like
I,
should
put
a
warning
like
no
vm's
were
harmed
in
this.
A
That's
fine.
We
have
one
minute
left
I'll
just
talk
to
you
about
what
happens
so
I'll
get
I'll,
give
it
like
a
hot
minute
and
then
we
can
see.
Oh
man,
I
expected
a
lot
of
failures.
I
did
not
expect
that
one,
so
I
was
very
trusting
of
conference
Wi-Fi.
So
what
happens
in
this
case?
If
we
so
I
killed,
node,
2
and
node
3
leaving
node
1,
and
what
I'm
gonna
do
on
there
is
just
go:
docker
swarm
an
it
or
use
the
sed
command.
That's
the
equivalent
and
force
new
cluster.
A
It's
gonna
look
back
to
the
raft
log!
That's
stored
in
that
that
wall
directory
that
we
talked
about
before
and
just
kind
of
revive
itself
understanding
that,
then
there
were
supposed
to
be
5,
elastic
searches,
replicas
running
and
it's
gonna
revive
those
five,
but
we
still
have
kind
of
some
cleanup
to
do
because
we
didn't
actually
gracefully
terminate
the
other
node.
So
we
have
to
get
rid
of
those
two
old
nodes
that
were
in
our
consensus
group
manually
or
scripting,
or
whatever
your
your
tool
of
choices
and
then
re-add
new
managers.
A
If
I
want
more
than
one,
let
me
try
this
one
more
time
before
admitting
defeat.
That's
fine,
so
I
will
leave
that.
Actually,
this
is
always
up
because
I
have
a
really
bad
habit
of
buying
domain
name,
so
I
own
consensus,
tech
group.
It's
like
I,
really
love
raft.
So
this
is
up
in
perpetuity.
You
can
check
this
out
or
secret
lives
of
data.
I
will
let
you
all
go
to
lunch,
but
I'm
happy
if
you
want
to
come
up
to
the
stage
after
to
answer
any
questions.