►
Description
Configuring and Deploying Stateful Applications with PetSet - David Oppenheimer and Prashanth Balasubramanian, Google & Clayton Coleman, Red Hat
Kubernetes 1.3 introduced the PetSet abstraction, which represents a collection of Pods with unique identities represented by an index, along with associated persistent storage and network identities. PetSet makes it easy to run stateful services on Kubernetes, including clustered applications like ZooKeeper and Redis. In this talk we discuss the design and implementation of PetSet, show how to use PetSet to deploy some common off-the-shelf stateful applications, and discuss our future plans for PetSet. We also describe “init containers,” which allow you to set up the user environment of a pod before the main container starts executing. We demonstrate how you can use this feature to configure the relationship between master and peers in a clustered application that is managed by a PetSet, and how new replicas are automatically configured when the application scales up.
About
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
All
right,
so
my
name
is
Clayton
Coleman
I'm
architect
for
openshift
I've
been
at
kubernetes
community
contributors
since
near
the
very
beginning,
and
today
I'm
going
to
talk
about
staple
sets.
Do
you
guys
start
about
that?
You
guys
came
to
this.
You
probably
saw
a
different
title
on
the
schedule
and
I
talked
about
staple
sets.
I
think
it's
really
important
that
we
get
this
out
of
the
way
upfront.
A
This
talk
is
absolutely
100%
not
about
pets
is
about
stateful
apps
they
just
recently
we
changed
the
name
of
pet
sets
to
stateful
sets,
and
there
was
a
big
discussion
in
the
community,
and
the
idea
here
is
that
the
the
general
idea
of
a
pet
is
this
something
you
care
about,
and
so
we're
just
gonna
be
a
little
bit
more
precise
and
we're
gonna
talk
about
state.
But
during
this
talk,
I'm
absolutely
guaranteed
that
I'm
gonna
say
pet
sets
so
I
say
pet
sets
or
stateful
sets.
I
mean
the
same
thing.
A
A
Everything
in
kubernetes
is
about
a
pattern
that
makes
applications
easier
to
run,
administer
and
keep
running,
and
so
keeping
running
is
really
important.
If
you
have
an
application,
that's
downs,
not
much
of
an
application.
The
real
world,
unfortunately,
is
not
so
nice
to
us.
Kubernetes
is
supposed
to
be
flexible
and
able
to
recover
from
failures.
A
There's
two
solutions
for
dealing
with
failure:
these
are
pretty
common
patterns.
The
first
one
is
detect
that
something
bad
actually
happened
and
restart
the
application
move
it
someplace
else.
This
is
often
called
mean
time
to
recovery.
If
you
plan
for
this,
you
can
actually
be
pretty
effective.
The
other
option
is
running
multiple
copies,
which
kind
of
depends
on
the
first
one.
I'll
talk
a
little
bit
more
about
that
in
a
minute,
so
recovery
crew
Bernays,
is
actually
pretty
good
at
recovery.
This
is
one
of
the
fundamental
principles
of
how
kubernetes
is
built.
A
The
idea
of
grenades
is
you
say
this
is
what
I
want
to
run
and
the
infrastructure
tries
to
keep
it
running.
So
if
you
only
need
three
nines
of
availability,
99.9%
availability,
that's
eight
hours
of
downtime
a
year
more
or
less
than
less
I
did
the
math
wrong
and
for
a
lot
of
simple
applications.
That's
a
lot
of
downtime
right
I
mean
eight
hours
you
it
takes
you
three
or
four
minutes
to
restart,
and
you
have
less
than
10
or
15
outages
a
year,
you're
well
within
that
budget.
A
So
kubernetes
is
really
good
at
two
things.
Restarting
things
that
crash,
as
well
as
detecting
nodes
that
have
gone
down
or
they
have
problems,
and
in
this
case
in
kubernetes,
if
you
detective
something's
wrong,
there's
a
process
that
will
come
along
and
it'll
say
you
know
what
this
pods
not
working
anymore.
I
want
to
stop
it
and
another
one
will
get
started
somewhere
else
on
the
cluster,
but
that's
just
for
simple
applications.
If
you
start
building
more
complex
applications
that
have
more
dependencies,
you
know
you
might
depend
from
your
front-end
on
a
web
service.
A
That
also
depends
on
a
database
and
if
you
start
adding
more
and
more
web
services
because
you're
building
a
micro
service
infrastructure.
What
actually
happens
is
that
the
margin
for
error
tends
to
be
a
lot
smaller.
It's
not
just
if
your
application
fails
it's
if
any
of
the
dependencies
fail
and
you
can't
recover
from
that
so
raqudii
and
recovering
can
work,
but
the
bigger
and
more
complex
your
applications
get
the
more
the
less
margin
you
have
for
transient
failures.
A
A
The
moment
you're
dealing
with
something
that
actually
matters
that
has
to
be
consistent,
so
qu
Bernays
is
really
good
at
replication
in
kubernetes
there's
the
first
primitive
above
a
pod,
which
is
something
that
runs
containers,
was
a
replica
set,
and
in
this
case
you
say,
I
want
one
of
these
and
it
goes
in
runs
in
the
cluster.
So
imagine
that
Harrison
Ford
is
running
on
your
cluster
somewhere
and
then
you
say
you
know
what
I
want
to
have
some
backup.
A
A
You
know
you
tell
everybody
to
go
store
some
data
and
the
first
Harrison
Ford
says:
yep
I
got
it
and
suddenly,
if
you
go
back
and
ask
all
the
Harrison
Ford's,
what
the
value
of
a
vehicle's
one
is,
you're,
probably
not
gonna
get
a
very
good
answer,
and
then
you
know
this
is
this
should
be
no
surprise
to
anybody.
This
room
but
I
think
it's
building
up
to
why
we
think
the
replication
isn't
enough.
So
as
part
of
the
design
process
for
kubernetes,
we
started
with
pods
we
build
on
top
of
them.
Replica
sets.
A
Replica
sets,
has
just
create
three
copies
and
we
know
that
we
want
to
help
people
deal
with
states.
State
is
a
fundamental
part
of
the
applications.
We
run
it's
not
just
about
web
services.
It's
about
databases,
it's
about
message,
queues,
it's
about
applications
that
write
data
to
disk
and
then
send
that
to
other
places,
and
so
it's
part
of
the
research
for
this.
We
we
kind
of
geeked
out
on
data,
there's
been
a
long
discussion.
This
is
not
a
new
topic
in
the
community.
There's
an
immense
amount
of
research
out
there.
A
There's
immense
amount
of
types
of
systems
that
actually
store
state-
and
this
is
just
a
portion
of
this
spreadsheet,
but
early
on,
we
went
through
and
kind
of
tried
to
catalog
a
lot
of
the
patterns
that
we've
seen.
You
know
on
the
far
left.
You
might
have
no
sequel
and
sequel
databases.
The
trade-offs
here
usually
are
kind
of
the
ones
that
we're
all
kind
of
familiar
with.
So
if
you've
heard
of
the
cap
theorem,
you
get
to
make
a
choice.
A
When
you
have
distributed
systems,
you
can
focus
on
consistency
or
you
can
focus
on
availability
in
the
event
that
something
is
partitioned,
and
so
we
kind
of
we
tried
to
go
through
this
thing
and
say
what
are
the
common
ideas
that
come
out
of
all
of
these
types
of
software
and
again
these
are
just
components
that
people
tend
to
reuse,
there's
an
almost
infinite
number
of
applications
that
actually
have
state
that
exist
in
the
real
world
that
aren't
represented
here.
So
a
classic
example
might
be.
A
If
you
have
a
large
je
application,
that's
been
taking
over
for
ten
years
quite
happily
now,
and
you
have
a
session
store.
That
might
be
using
Java
serialization
to
do
in
memory
copy
across
those
machines.
That's
a
stateful
set
that
sent
the
session
data
has
to
be
replicated
across
the
cluster,
their
data
and
if
one
of
those
machines
goes
down
in
your
state's
gone
a
user
who
has
you
know
a
shopping,
cart
or
a
web
order,
or
something
like
that
comes
back
in
and
suddenly
their
cards
gone?
A
You
know
these
are
older
problems,
but
they
still
carry
through
today.
So
we
looked
at
all
this
data
and
I
think
it's
important
that
I
stop
here
and
say
if
you're
running
stateful
things
on
kubernetes
today
they
do
work,
it's
possible
today
to
set
up
a
replication
controller
or
a
replica
set
and
use
a
persistent
volume.
And
if
you
set
things
up
right,
your
data
is
going
to
be
preserved.
If
a
failure
happens,
it's
going
to
take
over,
but
our
goal
is
not
just
to
make
it
possible,
but
to
make
it
easy.
A
If,
if
you
have
a
workload,
if
qu
Bernays
is
the
goal,
if
the
Careers
goal
is
running
applications
and
some
applications
have
state
it's
kind
of
silly
to
say,
you
know
what
you
can
run
state,
but
there's
all
these
rough
edges,
so
we're
not
going
to
help
you
so
the
stateful
said
is
taking
some
of
the
lessons
is
going
through
the
patterns
that
we
see
and
boiling
them
down
to
a
couple
of
core
concepts.
So
staple
said
is
a
little
bit
like
a
replica
set.
A
It
replicates
software,
but
it
adds
two
things:
the
idea
of
identity
and
of
tracking
State
and
States
a
real
catch-all
term
for
the
sake
of
argument
here
it
really
means
access
to
storage
or
access
to
something
that
is
persistent
beyond
an
individual
instance.
So
pods,
as
you
may
be
aware,
are
built
on
top
of
containers
which
are
ephemeral
so
in
kubernetes,
if
you
schedule
a
pod
and
that
machine
gets
wiped
off
the
face
of
the
planet.
If
your
data
only
exists
on
that
machine
data
is
not
coming
back.
A
Kerber
days
makes
it
easy
to
attach
persistent
volumes
to
your
machines
that
might
be
network
attached.
You
might
be
using
a
cloud
provider
with
AWS
or
GCE,
and
you've
got
persistent
disks
where
the
cloud
provider
has
done
all
the
hard
work
of
letting
you
pretend
that
data
doesn't
matter,
and
these
machines
are
all
fungible,
but
joy
might
also
be
running
on
bare
metal,
using
NFS,
I,
scuzzy
fibre,
channel
Gluster
Ceph,
any
of
the
other
plugins
that
people
have
been
adding
and
that
state
whether
it's
on
persistent
volumes
or
not.
A
We
want
to
make
it
easy
to
reason
about
so
because
state
is
important,
because
state
means
data
safety
first,
as
a
user
of
kubernetes.
If
you
use
the
system,
it
doesn't
help,
you
understand
what
it's
doing
and
what
the
failure
modes
are
and
how
you
can
hurt
yourself.
It's
not
a
very
good
system,
all
right,
if
you
trust
your
data
to
the
system,
but
you
can't
understand
and
expect
it
to
work
the
way
you
want
designed
for
the
administrator
or
the
application
author
or
the
human.
It's
not
a
very
good
system.
A
So
we
kind
of
formalized
the
three
laws
of
staple
sets
may
be
familiar
to
some
of
you.
A
staple
set
may
not
injure
your
state
or
through
inaction,
allow
state
to
come
to
harm
formalization
of
that
first
principle,
which
is
do
not
lose
data
that
you
trust
to
the
set,
and,
yes
sure
you
know
this
is
programming.
Things
are
hard.
People
always
make
mistakes,
but
first
and
foremost
we
shouldn't
lose
data
or
by
not
helping
or
telling
the
user
allow
that
data
to
be
lost.
A
A
second
you
know
take
orders,
do
what
you're
told
unless
it
conflicts
with
the
first
order,
so
we
want
to
failsafe.
So
if
you
say
do
something
stupid,
we
want
to
make
sure,
there's
enough
patterns
and
things
built
into
stateful
sets
that
say
that
was
stupid.
I'm
not
gonna.
Allow
you
to
do
it
or
if
you
do
it,
here's
why
this
is
going
to
cause
you
a
problem
and
I'm,
really
sorry
but
I'm
about
to
lose
your
data.
A
A
We
don't
have
a
prosthetic,
positronic
brain
or
anything
behind
this,
although
I
wish
we
did
so
I'm
going
to
start
with
a
really
quick
demo
and
then
come
back
and
talk
through.
What's
going
on
in
the
demo,
so
I'm
going
to
show
I'm
gonna
show
you
not
having
access
to
wireless,
but
so,
while
this
password.
A
A
So
I'm
gonna
show
you
guys:
openshift
I
work
on
open
shift
up
and
shift
is
a
distribution
of
kubernetes.
We
had
a
bunch
of
other
stuff
around
kubernetes
in
a
kind
of
opinionated
way,
but
ultimately,
under
the
covers
it's
still
raw
Cabernet,
so
I'm
running
open
shift
1/4,
which
is
built
on
top
of
kubernetes
1/4.
It's
pretty
easy.
We
try
to
keep
the
numbers
the
same,
except
when
we
ship
it
to
people,
and
then
we
add
3
for
some
arbitrary
reason.
A
So
an
open
shift,
a
Cooper
net
or
an
open
shift
project
is
just
a
kubernetes
namespace.
So
what
I'm
gonna
do
is
I'm,
going
to
take
a
zookeeper
example
written
by
prashanth,
who
could
not
be
here
today
because
he
made
me
do
it.
Prashanth
is
Sean
put
together
this,
as
we
were
doing
the
Alpha
work
for
pet
set.
So
what
I'm
doing
is
I've
got
a
I've
got
a
list
of
kubernetes
resources
and
I'm,
just
adding
them
in
and
creating
them
here.
A
A
We're
gonna
see.
I
talked
a
little
bit
about
the
idea
that
we're
going
to
be
predictable
and
help
you
not
mess
up.
So
one
of
the
things
that
people
tend
to
get
wrong
is
they
start
a
whole
bunch
of
stuff
at
once
and
they
all
try
to
do
things.
It's
not
necessarily
easy
to
reason
about
the
order.
Those
things
happen
in
so
the
way
that
pets
that
works
is,
we
start
the
first
one
up
and
then
we
walk
through
the
set.
A
So
you
kind
of
saw
a
little
bit
of
this
in
the
UI.
So
what
is
the
staple
set
or
pet
set?
A
staple
set
is
designed
to
stamp
out
identical
copies
of
a
pod,
and
this
is
a
pod
template.
So
if
you've
ever
created
a
replica
set
or
a
daemon
set
in
kubernetes,
this
should
be
really
familiar.
It's
just
you
know.
I
want
to
go,
create
an
open,
JDK
container
and
I
want
a
mount
in
a
data
directory.
A
The
special
thing
about
Pet
sets
and
staple
sets
is:
we've
talked
about
identity.
The
identity
is
the
idea
that
even
as
individual
pods
come
and
go,
there
is
an
instance
of
the
software
running
that
thinks
it's
the
first
one
and
there's
an
instance
that
thinks
it's
the
second
one
and
there's
an
instance
that
things
there's
a
third
one.
If
one
of
those
fails
we're
gonna
use
the
name-
and
this
is
the
name
of
the
pod-
we're
gonna
use
this
as
the
thing
that
we
organized
on.
A
But
if
you
want
to
have
three
members
that
each
have
their
own
volume,
you'd
have
to
go,
create
three
replication
controllers.
Three
replicas
sets
and
then
go
fiddle
around
with
a
bunch
of
details
too,
because
you
can't
scale
them
up
so
in
kubernetes,
the
pet
set
adds
the
idea
of
a
volume
claim
template.
So
today,
in
kubernetes,
when
you
want
storage,
you
say:
hey
I
want
to
go,
get
one
gig
of
storage,
and
what
will
happen
is
at
creation
time,
instead
of
just
creating
this
one.
A
This
really
depends
if
you've
used
persistent
storage,
this
actually
really
heavily
benefits
from
having
dynamic
provisioning.
So
that's
a
feature
that
was
introduced
in
1001
for
in
a
slightly
better
form,
storage
classes
were
added
in
one
for
as
alpha
and
are
starting
to
come
out.
So
the
idea
of
dynamic
provisioning
is
you
just
say:
you
want
storage
and
if
you
happen
to
have
a
pretty
awesome
storage
provider,
they
just
go
off
and
they
create
a
volume
for
you
on
demand.
A
It
makes
it
a
little
bit
fiddly
to
demo
on
your
laptop
and
that's
something
over
time
will
address,
but
we're
starting
to
get
there,
and
then
it
has
to
be
possible
to
discover
those
other
ones.
So,
if
I
have
su
0
zu
1
+
zu
2,
almost
all
clustered
software
needs
to
figure
out
how
to
talk
to
those
other
members
of
the
set,
and
so
in
kubernetes.
You
know
I
said
the
word
discover
in
kubernetes.
A
The
answer
to
service
discovery
is
services,
so
we've
added
some
features
to
services
which
I'll
talk
about
in
a
little
in
a
few
minutes
that
make
it
easy
for
the
individual
members
to
go
find
each
other,
but
also
for
someone
who's.
Talking
to
this
application
to
either
say
you
know
what
I
don't
care
about
which
zookeeper
instance
I'm
talking
to
so
want
to
talk
to
a
zookeeper
or
to
go,
find
the
individual
members
of
this
set
by
name
and
then
it's
really
important.
A
Safety
first
would
never
allow
two
pods
with
the
same
name,
to
run
at
the
same
time.
I'll
come
back
to
that
in
a
minute,
but
the
idea
here
is
that
if
two
people
think
they're
the
same,
you
could
have
pod
0
or
zoo
zero
running
one
point
of
cluster.
If
you
could
get
another
one,
that
things
said
zero
and
the
two
of
them
are
talking
to
different
members
of
the
set.
You
have
the
dreaded
split
brain
right.
Both
of
those
guys
are
participating.
A
A
A
If
you
ask
for
the
zookeeper
set,
what
you're
guaranteed
to
get
is
you're
never
going
to
get
4
members
here,
but
you
could
get
the
possibility
that
if
one
of
these
gets
replaced
and
comes
back,
you
might
get
another
one
in
between
another
feature.
That
we
use
is
that
we
added
in
Cooper
Nettie's
is
the
idea
of
a
knit
container.
So
when
we
started
Cooper
neighs,
you
have
pot
or
you
have
containers
in
a
pod.
You
can
run
one
two
three
and
infinite
number,
not
actually
an
infinite
number
of
containers.
A
At
the
same
time,
those
all
start
up
at
the
same
time
and
move
forward.
They
can
share
volumes.
They
can
talk
to
each
other
over
the
local
network,
but
it
was
really
important
for
us
to
be
able
to
separate
out
the
responsibility
for
separate
out
the
responsibility
for
setup
from
actually
running
your
container.
So
if
you
have
a
container
that
runs
zookeeper,
there
might
actually
be
things
that
you
need
to
do
ahead
of
time.
You
want
to
generate
a
config
file.
A
You
want
to
go
find
out
who
the
other
members
are
and
write
that
config
file
out.
In
the
example
that
I'll
show
Sean
actually
got
really
clever,
and
he
said
you
know:
I,
don't
even
really
want
to
maintain
docker
image
that
has
zookeeper
in
it.
I'm
just
gonna
use
the
JDK
and
I'm
gonna
download
from
a
particular
URL.
The
version
of
zookeeper
and
I'm
gonna
put
it
on
disk,
and
then
all
the
other
containers
in
the
pod
are
going
to
use
that.
A
So
there's
a
number
of
ways
that
you
can
use
the
net
containers
it's
not
coupled
to
stateful
sets,
but
it
is
a
tool
that
can
be
used
to
separate
out
the
logic
in
your
pots.
If
your
application
needs
to
wait
for
another
service
to
come
up
like
if
your
Java
Enterprise
application
doesn't
really
like
it.
If
the
database
is
down
when
it
starts
up,
and
it
gives
a
big
nasty
error,
you
can
write
a
bash
loop
in
the
command
arguments
here.
A
Yeah,
so
the
question
is:
is
there
eh
getting
an
IP
address?
Yes,
so
these
are
the
pod
IP
addresses
yeah
and
we'll
talk
about
that
at
the
very
end.
There's
an
infinite
number
of
features
that
complex
clustered
systems
need
to
work,
and
one
of
the
focuses
has
been
getting
the
core
ideas
in
place.
These
are
pod
IP,
spot
IPS
are
fungible
because
pods
can
come
and
go
and
depends
on
your
network
down
the
road.
There's
been
a
request
for
the
ability
to
have
virtual
IPS
for
each
member.
Just
it's
not
possible.
A
Now
we'll
come
back
to
it.
We
definitely
know
it's
a
problem
so
to
go
back
to
the
demo.
Let's
hope
that
everything
came
up.
Oh
my
gosh
I
got
lucky
so
I've
got
my
three
members
running
as
you
can
see,
they
all
have
the
same
names
that
I
was
talking
about
the
demo
on
the
overview
page
I
can
see,
I
have
three
pods
and
they're
listening
on
two
ports.
So
there's
nothing
really.
A
As
far
as
kubernetes
and
openshift
is
concerned,
the
pods
are
basically
all
the
same.
All
we
saw
was
that
they
were
part
of
the
same
staple
set
and
as
we
group
them
together
in
the
UI,
the
service
didn't
change.
All
that
much.
Let
me
jump
on
the
command
line,
so
I'm
going
to
run
the
demo
and
we'll
see
if
this
works.
So
let's
hope
so.
This
is
kind
of
Prashant's,
simple
hello
world
for
zookeeper.
A
So
what
this
does
is
it
connects
using
exec
into
one
of
the
members
of
the
zookeeper
cluster,
so
in
this
case
I
think
he
connected
to
zu0
and
what
he
did
was
he
just
made
a
simple,
create
call
that
went
in
created
a
single
key
in
the
zookeeper
cluster,
and
then
we
connect
to
the
second
instance
and
I:
don't
know
that
it
actually
prints
the
name
out
down
here.
I
see
if
I
can
find
it.
A
A
The
idea
is
to
put
enough
primitives
in
place
that
this
is
easy
to
do
impossible
to
do
and
possible
to
do
safely,
but
it
doesn't
do
it.
It's
not
magic
right,
there's
no
magic.
Here.
This
is
saying
we
have
a
basic
idea.
We
want
to
create
multiple
copies
of
something
cou
Bernays
is
gonna.
Go
handle
all
the
lifecycle
of
these.
If
the
machine
goes
down,
Cabrini's
gonna
take
care
of
it.
It'll
bring
back
a
new
one
if
it
can
all
the
existing
kubernetes
primitives
work,
but
zookeeper.
A
We
had
to
do
a
little
bit
of
glue
code
to
set
that
zookeeper
cluster
up
and
that
glue
code
is
actually
a
really
important
part,
because
each
of
those
clustered
bits
of
software
I
showed
those
aren't
simple
systems
right.
If
you
ever
run
an
elasticsearch
cluster
or
Kafka
or
zookeeper
SCD
or
my
sequel,
met
mexico,
master/slave,
sharded,
cluster
or
Hadoop
or
HDFS,
there's
lots
of
things
that
can
go
wrong.
A
I
think
I
never
want
to
leave
people
with
the
impression
that
stateful
sets
are
gonna
just
magically
make
problems
go
away,
but
they're
trying
to
provide
a
rock-solid
primitive
that
people
can
build
higher
level
logic
on.
If
you
have
ten
elasticsearch
clusters,
you
may
still
need
to
do
some
actions.
Administrative
ly,
to
keep
those
closures
up-to-date,
do
rolling
updates
and
handle
those
tools
and
overtime.
A
The
goal
is
to
get
the
community
to
be
able
to
help,
pull
together
examples
and
tools
that
work,
but
I
suspect
that
for
the
next
couple
of
years,
when
you
look
at
these
things,
you're
going
to
still
have
systems
that
have
really
good
fundamentals.
You
can
depend
on
kubernetes
to
key
things.
Running
and
you'll
build
tools
on
top
scripts
systems
patterns
that
make
this
easier
or
they
make
it
easier
to
reason
about.
A
What's
going
on,
it's
not
going
to
magically
solve
all
problems,
and
so
I
I
would
caution
that,
if
you're
looking
at
staple
sets
today,
if
you're,
trying
to
use
staple
sets,
if
you're
coming
and
getting
involved
in
kubernetes,
you've
heard
about
these
amazing
things
that
can
do
all
of
these
wonderful
things.
For
you
take
a
deep
breath
step
back.
You
guys
know
the
systems
that
you
run
and
you
work
with
them.
A
A
If
I
started
up
three
of
those,
only
one
of
them's
gonna
come
up,
kubernetes
will
come
along
if
that
node
goes
down
the
cloud
provider
and
all
this
magic
will
actually
work
to
detach
that
volume
from
the
node
and
then
you'll
start
up
a
new
pod.
So
you
can
get
availability
that
way,
there's
a
trick,
and
so
today,
in
kubernetes,
if
a
node
goes
down,
I've
stops
responding,
stops
checking
in
we
evacuated
and
after
it's
been
evacuated
and
if
it
hasn't
responded
for
a
long
period
of
time.
A
We
say
you
know
what
we
don't
need,
those
pods
anymore,
so
we
go
delete
each
and
every
one
of
them
that
doesn't
mean
that
that
pod
stopped
so
the
beauty
of
distributed
systems
is
the
the
masters
have
no
idea
what's
happening
on
those
remote
systems.
Unless
they
can
talk
and
if
they
can't
talk,
it
doesn't
mean
that
the
world
is
proceeding.
Normally,
it
means
crazy
and
sane.
A
Things
are
happening
on
the
background
and
buyer
beware,
and
this
is
something
that
sis
amends
for
the
last
50
years
have
dealt
with,
and
we've
built
more
powerful
tools
to
deal
with
this
things.
Like
pacemaker
virtual
IDs
keep
a
live
D.
We
built
systems
that
work
together
to
detect
failures
and
to
help
administrators
reason
about
them.
A
So
if
the
pod
hasn't
been
stopped
because
for
whatever
reason
that
node
can't
talk
to
the
master,
but
it
can
still
talk
to
the
other
zookeeper
members,
if
we
created
in
a
new
pod
at
that
point,
we
would
go
and
actually
have
two
pods
running
at
the
same
time,
because
we
forced
delete
it.
We
said
zu0
gone
deleted,
oh
look,
there's
no
zeros
anymore!
I'll
go
create
another
one,
so
starting
in
1-5.
We
fixed
this,
and
this
was
all
my
fault
by
the
way.
A
So,
when
I
put
in
graceful
deletion
to
kubernetes
I
take
full
responsibility,
we
were
a
little
aggressive
about
going
in
cleaning
stuff
up.
I
was
brought
to
sanity
by
the
excellent
community
members,
including
some
people
in
this
room
who
I
won't
call
up
my
name
in
starting
in
queue.
Bernays
1-5,
and
this
is
really
important.
A
I
go
and
I
go
delete
the
the
pod
format
CD
or
from
the
API
server,
which
is
basically
the
same
thing
in
1-5
when
this
happens,
we'll
evacuate,
but
we
won't
go
and
force
delete
stuff,
so
it'll
still
be
there
in
the
API
you'll
still
see
zu
0
it'll
say
it's
unreachable
as
an
admin.
You
have
a
choice.
You
can
say:
I
know
that
the
node
this
hit
pod
was
on
is
gone.
If
you're
running
on
a
cloud
provider
like
GCE
or
a
TBS.
A
A
You
can
go
forced
believe
the
pod
and
the
state
will
settle,
will
pick
up
the
next
piece
and
take
over
it'll
go
create
a
new
pod,
but
otherwise
it'll
fail
safe,
fail
in
the
direction
of
causing
the
least
possible
damage
and
you'll
run
it
or
reduced
availability.
So
if
you
have
three
members,
one
goes
down.
A
You'll
have
two
that's
enough
for
zookeeper
to
keep
running,
but
your
failure
margin
will
have
been
degraded
and
obviously,
if
you're
running
kubernetes
today-
and
you
don't
have
an
alerting
system
that
tells
you
when
nodes
go
down,
you
should
add
an
alerting
system
that
tells
you
what
notes
go
down
this.
The
cluster
is
not
perfect.
The
real
world
is
a
crazy
ugly
place
and
bad
things
happen.
A
So
monitoring
understanding
how
the
system
is
working
is
critically
important,
and
this
is
a
part
that
we're
putting
in
just
to
make
sure
that
there's
a
there's
a
base
level
of
logic
that
people
can
reason
about
to
be
a
safe
as
possible.
Future
work
includes
I
talked
about
keep
alive,
D
and
you
know
concepts
like
shoot.
The
other
node
in
the
head,
which
sounds
really
traumatic,
and
we
probably
won't
call
it
that
when
it
goes
into
cooper
neighs,
because
that
might
also
be
somewhat
contentious,
no
shooting
in
kubernetes.
A
The
idea
is
more
broadly
called
fencing.
Node
goes
down,
you
say:
I
can't
necessarily
guarantee
that
I'm
going
to
be
able
to
shut
that
note
down
cleanly,
but
the
rest
of
the
cluster
can
act
to
limit
the
impact
that
node
might
have
on
the
rest
of
the
cluster.
So
this
is
really
common.
In
the
storage
space
used
to
be
storage
systems,
don't
didn't,
have
very
good
controls
over
who
can
talk
to
the
storage
system.
A
A
It
could
be
something
that
ships
was
part
of
the
box
that
communities
for
bare
metal
systems
they
can
come
in
and
say:
I've
got
failure
detectors
when
I
see
something
go
wrong.
I
can
help
kubernetes
resolve
this
partition
and
the
fencer
might
be
someone
who,
if
they're,
absolutely
sure
that
machines
gone,
they
can
take
on
immediate
action.
So
this
is
a
lot
of
future
work
here.
The
goal
is
to
build
a
sane,
reasonable
system
that
people
can
layer
on
top
of
staple
sets
will
enter
beta
and
1/5.
A
It's
really
important
to
us
that
you'll
be
able
to
reason
and
trust
that
the
stateful
set
is
not
going
to
lose
data,
which
is
why
1/5
would
the
change
I
talked
about
is
going
to
be
when
it
goes
into
beta,
we'll
probably
have
a
fairly
long
beta.
We
want
to
get
feedback
examples,
features
that
are
missing.
We
need
the
ability
to
update
staple
sets,
I'm
sorry
to
like
take
away
more
of
like
the
pretty
pink
bow
around
staple
sets.
A
If
you
shut
down
a
bunch
of
nodes
and
random
or
a
bunch
of
pods
in
random
order,
and
all
that
data
gets
lumped
on
to
one
box
and
people
are
still
slamming
data
at
that
system,
you
might
your
whole
cluster
may
come
over.
So
it's
not
a
trivial
problem.
We
want
to
make
more
extensive
tools
for
operating
sets.
A
That
makes
it
easy
to
stand
up
at
CV
clusters
or
Prometheus
cluster
and
handles
that
sort
of
resolution
of
what
happened
so
Etsy
D
has
its
own
particular
foibles
about
how
its
initialized
and
how
you
add
and
remove
new
members
and
the
idea,
the
operators
you
formalize,
some
of
that
knowledge
into
code,
but
one
of
the
challenges
comes
is
well.
Does
that
generally
apply
to
elasticsearch?
A
No,
and
so
one
of
the
things
we
want
to
work
with
in
the
community
is
finding
the
points
of
commonality,
above
this
base
layer
for
things
like
operators
and
scripts
and
tooling,
and
hooks
that
we
might
add
back
into
staple
sets
over
time
so
and
we
want
to
add
tests
and
documentation.
Obviously,
if
it's
not
easy
to
go,
try
this
out
and
understand
how
it
works.
It's
not
much
good
to
people.
A
Local,
persistent
storage,
so
I
talked
about
persistent
volumes
if
you
have
dynamic
provisioning,
but
if
you're
running
on
bare
metal-
and
you
don't
have
this
massive
cluster
of
auto
provisioning
stuff,
local
persistent
storage
is
one
way
not
the
only
way
of
making
it
a
little
bit
easier
to
do.
I
create
three
members.
Instead
of
having
to
go
in
provision
asset
block
device
for
that,
I
would
like
to
be
able
to
create
a
local
volume.
That's
attached
to
that
machine
and
I
have
to
understand
the
trade-offs
there
right.
A
This
is
a
that
data
is
located
on
that
machine.
Somebody
eventually
would
have
to
go
make
it
move.
You'd
have
to
deal
with
downtime
we're
not
quite
there
yet,
but
it's
something.
That's
on
the
roadmap
for
cube
that
enough
people
need
local,
persistent
storage,
but
it's
gonna
be
really
important
to
do.
You
can
emulate
this
a
lot
of
ways
today,
but
we
want
to
make
it
easier
in
the
box
and
as
the
agenda
noticed,
software
that
needs
stabled.
A
Ids
are
stable,
IPS
for
each
member,
so
cassandra
has
a
particularly
nasty
habit
of
taking
those
nice
generic
DMS
dresses.
We
give
it
and
then
turning
of
that
into
an
IP
address
than
sticking
in
a
database.
So
some
of
the
work
that
we
talked
about
it
is
possible
to
build
a
script.
On
top
of
that
watches
the
Cassandra
cluster
and
goes
and
updates
the
Cassandra
member
said
we
want
to
explore
solutions
with
a
community,
but
probably
in
the
short
term.
We
have
no
good
answer
for
this
I'm.
Sorry,
it's
all
my
fault.
A
There
is
no
template
expansion,
and
so-
and
this
is
this-
is
a
so-
the
the
the
question
was
there's
no
template
expansion
in
that
template,
template
explain
tune
is
really
really
really
really
really
hard.
Probably
there's
Googlers
in
the
room
that
can
talk
your
ear
off
about
all
the
foibles,
about
the
ways
that
config
management
systems
fit
together
and
the
different
trade-offs
they
make.
We
in
favor
of
starting
down
that
path,
we
said:
well,
let's
take
a
step
back.
A
What
are
the
two
things
that
we
can
provide
on
the
Box
IP
are
stable
identity
and
state,
it's
still
possible
to
use
any
config
system.
You
want
to
go
generate.
Helm.
Charts
should
support
this
today,
because
helm
charges
just
deal
with
API
objects,
and
so
what
we'd
like
to
do
is,
as
that
discussion
evolves,
kind
of
keep
it
orthogonal.
Nothing
in
stateful
sets
depends
on
template
expansion
and
because
we
try
to
provide
that
base
primitive
and
it
there's
actually
no
guarantee
that
a
stateful
set
is
just
a
single
staple
set.
A
So
everything
that
works
on
a
pod
today,
including
node
in
turf
in
'ti,
should
work
today.
So
all
you
have
to
do
is
set
the
beta
annotation
for
intra
pod
affinity
and
it'll,
spread
them
automatically,
in
fact,
because
every
staple
said
basically
requires
a
service.
If
you
have
that
scheduling
predicate
set,
which
it
is
by
default,
I
still
think.
Yes,
thank
you.
Tim,
that's
set
by
default.
All
members
will
land
on
another
node
unless
there's
only
one
node,
so
I'm
running
on
my
laptop
here
in
the
demo,
I
was
running
in
a
VM.
A
A
Yes,
oh
and
I
didn't
quite
show
it,
but
the
stateful
set
always
goes
from
lowest
index
the
highest
index.
If
you
deleted
zero,
it
would
try
to
go
and
recreate
zero.
There's
still
a
lot
of
wrinkles
here,
the
documentation
that
I
alluded
to.
We
want
to
make
it
easier
to
describe
exactly
what
you
can
rely
on
and
show
examples.
I
would
say
today
the
ordering
might
be
enough,
but
it's
easy
to
back
yourself
up
again,
like
the
hard
part
about
all
this
is
it's
distributed
systems
which
we
can't
magically
solve.
A
So
what
we
try
to
do
in
the
staple
set?
Is
we
try
more
than
anything
else,
to
be
consistent?
First,
we
will
never
actively
introduce
inconsistency
and
so
working
through
how
you
would
apply
that
inside.
You
know,
for
instance,
in
zookeeper
we
will
never
create
more
than
three
and
we
will
never
create
two
before
the
first
one
or
never
could
zero
indexing
I
really
wish.
We
had
done
zero
indexing.
This
was
my
personal
beef,
but
I
was
overridden
on
that.
A
So,
no
and
part
of
the
reason
for
that
is,
you
can
actually
and
there's
been
adjust.
There's
actually
an
active
just
a
PR
discussion
on
this.
If
you
put
the
cardinality
as
a
downward
API,
and
then
you
change
that
cardinality
and
then,
if
you
can't
reason
about
so
example
here
as
I
start
with
three
zookeeper
nodes
and
then
I
scale
up
to
four
I
scale
up
to
oh
sorry,
I
scale
up
to
seven
I,
add
four
more.
A
If
you
hard-coded
three
in
your
initial
template,
just
as
an
environment
variable
or
whatever,
then
you
could
have
formed
an
initial
quorum,
we
would
have
never
created
more
if
you
hard-coded.
If
that
changed,
we
would
have
to
go,
delete
zero
one
and
two
and
ensure
that
they
observe
it.
Otherwise,
when
we
create
those
four
and
it
so
the
scenario
would
be
the
first
three
get
partitioned
before
the
before
we
detect
it
and
those
new
four
come
up.
A
That's
actually
a
majority,
and
if
they're
not
carefully
written
to
deal
with
the
fact
that
there
may
be
another
majority
in
the
cluster
when
you
resize
them,
you
could
end
up
breaking
it,
and
this
is
the
this
is
the
general
challenges.
Many
of
these
systems
have
innate
tools
for
handling
membership
change
that
require
certain
set
of
guarantees.
There
has
to
be
a
majority
at
all
time
to
add
new
members,
and
so
one
of
the
things
that
we're
trying
to
be
very
cautious
about
is
allowing
people
to
accidentally
introduce
initialization,
splits
and
the
documentation
there's.
A
All
of
this
is
in
people's
heads
or
in
pr's
or
discussions.
We
actually
want
to
have
a
really
good
document
that
says:
here's
how
you
can
safely
reason
about
it
and
the
guarantees
we
provide.
It
would
be
easy
to
add
that
in
the
template,
but
as
you
change
the
cluster
membership,
if
you
didn't
treat
your
own
internal
consensus,
it's
possible
to
get
split
brain
sir.
A
So
I
can't
speak
to
Borg,
although
I
could
call
Tim
out
if
he
wants
to
talk
about
that
he's
hiding.
So
the
answer
is:
is
that
I
in
general
and
I
think
I
can
speak
for
everybody
in
the
crew?
Bernays
project
is
distributed,
systems
are
hard
they're
not
going
away,
and
rather
than
hide
from
that,
what
we
want
to
build
is
a
system
that
can
eventually
deal
with
both
stateless
and
stateful
applications
correctly
safely
and
work
with
the
communities
to
build
it.
A
There's
no
other
path
forward
right,
because
everyone
who
needs
a
stateful
database
today
can
guarantee
you
they've
got
hacked
up
versions,
no
offense
to
anybody
out
here
who
has
hacked
up
versions
of
anything
we're
using
software.
That
does
all
these
things
today.
These
aren't
new
problems
as
much
as
possible,
trying
to
bring
communities
of
expertise
together
so
that
we're
all
solving
some
of
the
same
problems
and
everyone
can
benefit,
is
I.
Think
what
kubernetes
can
bring
to
this.
A
It's
it's
worth
trying
because,
where
there's
no
going
back
and
if
your
reasoning,
if
you're
every
single
person
out
there
who
cobbles
together
a
script
that
sets
the
bouzouki
per
cluster
on
a
table
us
or
on
GCE
or
runs
it
in
across
the
set
of
docker
containers,
those
are
really
hard
problems
somewhere.
Somehow
we
have
to
sit
down
and
make
sure
that
those
can
be
built
on
good
foundations
and
I.
Think
there's
solvable
problems.
It's
gonna
take
a
while,
like
it's
not
an
easy
problem,
but
I
think
it's
worth
solving.
A
A
There's
been
a
the
proposal
is
basically
yes,
that
you'd
create
a
persistent
volume
claim.
The
scheduler
would
put
you
on
a
machine
that
had
enough
storage
to
satisfy
your
request,
and
then
some
process
would
go
and
record
that
so
that
we
could
say
no.
You
are
not
allowed
to
run
on
a
different
machine
for
this
persistent
volume.
There's
a
couple
of
proposals
out
there
Tim
nodded
his
head
at
that
I.
Think
we'd
like
to
do
this
over
the
next
couple
releases.
A
A
So
there's
maybe
like
a
small
challenge
there
I
would
say
that
one
of
the
scenarios
that
we
have
talked
about
quite
a
lot
is
integrations
with
things
like
vault
and
other
distributed
secret
systems,
where
the
responsibility
for
getting
the
secrets
into
the
container
is
clearly
delineated
from
the
application
itself.
But
we
would
need
to
be
very
careful
about
how
that's
implemented.
A
Dns
is
provided
by
kubernetes,
identity
protection
is
provided
by
the
cluster,
and
then
the
DNS
reflects
what's
on
the
cluster
today.
That
is
a
strongly
consistent
feedback
loop
with
a
couple
of
small
wrinkles.
Our
goal
would
be
to
ensure
that
it
is
impossible
for
DNS
to
be
not
something
you
can't
reason
about,
but
knowing
the
part
of
DNS,
you
could
also
look
at
the
endpoints
array
that
backs
CNS
and
fetch
that
data,
because
that's
stored
in
sed
today.
A
That's
true,
and
the
one
of
the
ideas
here
would
be
that
bootstrapping
of
these
clusters
is
usually
the
first
part
and
then
the
members
try
to
connect
the
masters
anyway,
when
the
members
try
to
connect
the
masters,
if
you've
built
a
system
that
is
consistent
and
you
try
to
talk
to
something-
that's
been
split
brained
either
you
caused
the
split
brain
by
violating
the
cluster
safety
guarantees
or
you
you're
unable
to
join
that
cluster.
So
we
want
to
be
able
to
work
through
how
people
can
write
the
system
so
that
it
is
safe.