►
From YouTube: Keynote: Smooth Operator♪: Large Scale Automated Storage with... - Celina Ward & Matt Schallert
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Keynote: Smooth Operator♪: Large Scale Automated Storage with Kubernetes - Celina Ward, Software Engineer & Matt Schallert, Site Reliability Engineer, Uber
Managing stateful applications is tough. While Kubernetes excels in automated application management, more often than not, the state of these applications live outside of Kubernetes. What happens when users develop complex stateful software but still want to benefit from the automation and portability that Kubernetes provides?
In this talk, Celina and Matt share their experience creating an operator for a unique stateful workload: a sharded, replicated, multi-zone time-series database. They discuss the major shift in thought -- from the mental model of a 1:1 process/node relationship, to one far more adaptive to both machine and user behavior.
The audience will walk away with a framework for expressing their stateful workloads using Kubernetes primitives, and advice for navigating the difficult process of codifying innovative abstract ideas without over-engineering solutions.
To learn more: https://sched.co/Gsxn
A
Today,
we're
here
to
share
our
story
with
you
of
putting
em
3d
beat
on
kubernetes
Before.
We
jump
into
that,
though.
We
want
to
give
you
some
context
on
what
m3
DB
is
m3.
Db
is
part
of
m3,
which
is
ubers
open
source
metric
stack
m3
has
been
in
production
at
uber
since
2016
and
we've
built
it
in
the
open
since
day.
One
m3
DB
is
the
distributed
time
series
database,
that's
it
that
sits
at
the
core
of
m3,
alongside
our
other
components,
like
our
fault,
tolerant,
aggregation,
tier
and
distributed
query
engine
to.
B
A
When
we
first
built
m
3
DB
back
in
2016
operating,
it
was
pretty
simple.
We
ran
M
3
DB
in
two
data
centers
and
had
one
cluster
of
her
data
center.
Our
primary
use
case
at
the
time
was
real-time,
alerting
and
dashboard
introspection.
So
our
cluster
shared
a
static
configuration
fast
forward
to
2018.
However,
and
things
aren't
so
simple
for
one,
we
run
em
through
DB
in
more
data
centers
and
there
are
more
clusters
per
data
center.
The
clusters
themselves
are
larger
and
they
no
longer
share.
B
Here
are
some
features
that
make
em
dirty
be
interesting
to
work
with
then
30
based
doors,
real-time
metrics
from
all
of
our
services
and
hosts
so
upon
receiving
a
metric
and
30
be
shards.
The
data
into
its
respective
partitions,
each
shard
is
replicated
by
a
factor
of
three
which
are
then
placed
in
separate
failure.
B
Domains
the
replicated
environments
are
always
synched,
so
whenever
adv
nodes
are
added
removed
or
replaced,
the
replication
factor
in
isolation
group
is
honored
when
redistributing
the
shards
to
instances
in
terms
of
maintaining
and
30
be
the
main
pain
points
of
it
can
be
broken
down
into
two
cycles.
The
first
part
is
the
daily
reactive
cycle.
When
a
node
becomes
problematic
an
engineer
gets
paged,
then
they
would
manually
SSH
into
Ebers
house
to
reboot
provision.
Another
resource
or
decommission
the
resource
so
because
of
these
daily
happenings
continuous
multi-region
Environment
Management
is
a
must.
A
Our
first
step
was
to
survey
the
list
of
existing
tools
for
managing
applications
in
a
cloud
native
world.
Although
we
were
really
excited
by
the
rise
of
technologies
such
as
kubernetes,
we
wanted
to
make
sure
that
we
thoroughly
understood
our
problem
space
before
jumping
to
a
solution.
So
we
broke
down
what
we
were
looking
for
in
a
solution
into
a
few
core
requirements.
A
Our
first
requirement
was
that
whatever
we
chose
would
have
to
have
primitives
to
support
demanding
stateful
workloads
given
m3/d
DS
characteristics.
We
require
access
to
low
latency
storage
and,
given
our
customer
are
given
our
users
as
requirements,
we
require
access
to
the
storage
across
all
of
our
data.
We
have
internal
workloads
at
uber
that
randomly
query
large
time
windows
repeatedly
with
look-back
periods,
ranging
from
a
few
minutes
to
a
few
years
and
everywhere
in
between
to
give
you
a
sense
of
what
we
are
looking
for.
A
Let's
take
a
look
at
some
ways
of
managing
state
and
containerized
applications.
One
option
for
managing
our
state
would
be
to
just
not
manage
it
at
all:
an
m3
DB
instances.
Data
would
live
as
long
as
its
container.
This
might
technically
be
fast,
as
we
would
have
direct
access
to
the
host
disk,
but
it
otherwise
be
pretty
terrible
for
one
there's:
no
durability
of
the
data.
This
means
that
when
m3
DB
in
containers
died
or
restarted,
they
would
have
to
stream
terabytes
of
data
from
their
peers,
and
this
would
be
wildly
inefficient.
A
On
top
of
all
this,
when
M
3
DB
instances
are
streaming
data
they're
not
available
for
reads
which
would
have
dangerous
reliability
implications.
So
what
if
we
didn't
meet
to
choose
to
store
our
state
in
a
more
durable
manner,
using
something
like
a
remote
block
store?
This
is
a
pretty
popular
option,
but
unfortunately,
the
latency
guarantees
provided
by
a
remote
block
store,
didn't
really
match
what
we
were
looking
for.
A
M
3,
DB
stores,
all
of
the
monitoring
data
for
ubers,
global
real-time
marketplace
and
our
customers
expect
to
be
alerted
as
soon
as
there's
an
issue
for
us.
That
means
that
every
bit
of
latency
in
ingesting
and
serving
metrics
counts,
and
while
the
increased
durability
is
nice,
it
seemed
inefficient
to
pay
someone
else
to
replicate
our
data
slower
when
we
already
replicate
things
three
times
at
the
application
layer.
On
top
of
all
that,
a
remote
block
store
has
different
characteristics
across
different
cloud
providers
and
would
be
potentially
out
of
the
question
our
internal
data
centers.
A
Another
option
we
had
seen
for
storing
remote
state
would
be
to
use
something
would
be
to
deduplicate
the
data
and
store
it
on
a
remote
object
store
like
your
friendly
neighborhood
cloud
providers,
object
store,
such
as
s3
or
Google
Cloud
Storage.
Well,
this
is
a
little
bit
more
efficient
as
we're
only
paying
to
store
one
copy
of
the
data
we
still
weren't
comfortable
with
the
latency
that
it
introduced
for
real-time
alerting
and
additionally,
we
would
still
have
the
problem
where
we'd
have
to
stream
terabytes
of
data
as
containers
moved
around.
A
B
Second
requirement
was
to
have
our
solution
work
seamlessly
in
our
internal
data
centers
and
in
the
cloud
in
the
short-term,
for
better
or
for
worse.
Our
current
production
workloads
already
run
in
multi
cloud
environments
and
in
our
internal
data
centers
and
realistically
we
can't
just
ignore
one
or
the
other.
If
we
want
all
the
metrics
for
an
accurate
observability
picture
in
the
long
term,
developing
an
environment,
agnostic
solution,
better
prepares
us
for
the
future.
It
allows
us
to
dive
deeper
on
core
features
that
would
otherwise
be
considered
technical
debt
down
the
road.
B
Finally,
we
really
wanted
to
embrace
the
global
engineering
community.
We've
had
a
great
time
spending
our
efforts
open
sourcing
and
30b,
so
we
thought
it'd
be
pretty
silly
to
make
an
add-on
that
the
community
couldn't
take
advantage
of.
We
also
figured
other
smart
engineers
around
the
world,
find
themself
with
similar
challenges,
and
so
we
figured
there
must
be
innovative
ideas
out
there
to
leverage.
B
Meanwhile,
the
reality
of
the
situation
was
around
the
time
we
were
thinking
of
simplifying
and
30
B
management.
We
had
both
just
come
back
from
Curb
Khan
and
seen
how
others
had
leveraged
kubernetes.
For
the
workloads,
so
in
particular,
operators
caught
our
attention,
an
operator
is
a
set
of
application-specific
custom
controllers
or
custom
resource
definitions
with
direct
access
to
the
cube
API.
B
It
simplifies
the
lifecycle
of
cube
native
applications
from
packaging
deploying
and
managing
and
scaling
engineers
can
customize
rules
for
the
operator
to
observe
analyze
and
act
on
the
deployment
to
reconcile
state
and
here's
where
we
notice
similarities
in
the
pattern
with
our
MVP
lifecycle
to
explain
one.
Reconciling
state
there's
have
direct
access
to
the
cube
API
and
also
whatever
a
PS
you
choose
so
because
it's
a
custom
resource
definition.
It
extends
the
capabilities
of
kubernetes
as
an
endpoint,
in
whatever
context
you're
working
with.
B
In
our
case,
a
potential
operator
would
be
able
to
listen
and
observe
on
the
happenings
of
m30
B
and
its
relationship
to
kubernetes
and
makes
adjustments
wherever
necessary
according
to
the
declared
desired
state.
So,
given
our
culmination
of
requirements
combined
brewing
interest
in
operators,
we
decided
to
try
a
proof
of
concept
on
kubernetes.
The
big
thing
for
us
was
that
it
abstracted
away
machine
failures
and
management,
which
was
currently
the
biggest
human
operation
for
M
30
B.
A
While
we
were
designing
our
operator,
a
few
kubernetes
features
gave
us
increased
confidence
in
our
decision
and
were
critical
to
its
ability
to
meet
our
workload.
The
first
was
kubernetes
is
support
for
persistent
local
volumes.
In
addition
to
meeting
our
performance
needs,
persistent
local
volumes,
where
the
storage
abstraction
that
would
work
across
our
internal
data
centers
and
our
cloud
environments,
where
we
aren't
comfortable
using
network
storage
for
our
workloads.
A
Thinking
back
to
the
staple
properties
we
were
looking
for,
persistent
local
volumes
really
met
all
of
our
needs.
The
scheduler
is
aware
of
the
requirements
of
local
volumes,
so
kubernetes
keeps
our
containers
on
the
same
node.
If
it
can
for
durability
purposes,
the
schedulers
understanding
of
data
gravity
was
super
important
to
us.
Local
volumes
meant
that
we
could
still
have
access
to
fast
local
disks
and
that
there
wasn't
unnecessary
duplication
of
our
already
replicated
data.
A
Additionally,
kubernetes
is
support
for
node
affinity
and
the
safety
guarantees
provided
by
stateful
sets
were
really
helpful.
Stateful
sets
enabled
us
to
have
strict
ordering
or
strict
guarantees
around
the
ordering
of
pod
operations,
enabling
safe
deployments
and
cluster
upgrades
their
use
of
stable
pod
identity
and
sticky
storage
made
managing
our
cluster
topology
easier,
as
we
could
be
sure
that
instances
could
easily
rejoin
the
cluster
after
failures
and
restarts
if
they
had
the
same
plot,
identity
and
storage
attached.
A
Node
affinity
was
a
great
tool
for
us
to
provide
a
similar
abstraction
for
expressing
our
requirements
for
failure,
domains
across
our
data
centers
and
our
cloud
environments.
By
leveraging
node
affinity
and
stateful
sets,
we
could
pin
groups
of
M
3
DB
instances
to
zones
in
the
cloud
providing
highly
available
fault,
tolerant
clusters,
this
matched
what
we
were
used
to
in
our
internal
data
centers,
where
we
made
that
sure
that
M
3
DB
instance
were
evenly
dispersed
across
racks.
B
So,
with
this
deeper
understanding
of
our
requirements
and
the
confidence
that
kubernetes
would
be
able
to
support
our
workloads
without
making
performance
or
safety
requirements,
we
set
out,
building
and
deploying
a
proof-of-concept
operator,
and
we're
really
excited
to
show
the
results
of
that
experience
with
you
putting
all
this
together.
We
leverage
these
features
to
build
and
deploy
our
very
own
M
30
B
operator.
You
can
check
it
out
right
now
and
github.
B
This
operator
currently
handles
a
subset
of
ubers
production
and
PDB
workloads
on
GCP
on
gke
from
kubernetes
perspective,
we
started
with
our
clusters
running
in
the
cloud
to
lower
the
barrier
to
entry
for
kubernetes.
However,
from
an
MDB
perspective,
it
is
trickier
for
us
to
manage
clusters
in
the
cloud,
because
we
don't
have
all
the
original
tooling
that
we
originally
started
with
in
our
own
data
centers.
So
how
do
our
engineering
efforts
ultimately
help
our
team?
Now
when
a
node
is
on
fire?
Our
operator
is
notified
instead
of
an
engineer
on
page
or
duty.
B
The
operator
communicates
with
kubernetes
as
to
which
action
it
should
take
in
order
for
the
node
to
become
healthy
again
in
terms
of
managing
our
multi
region
environment.
Our
plans
are
now
themed
at
a
higher
level.
Instead
of
having
people
resize
clusters
for
the
week,
we
can
have
one
engineer
simply
make
a
configuration
change
to
adjust
for
the
appropriate
amount
of
resources.
We
are
actively
continuing
to
lessen
the
need
for
capacity
planning
meetings
as
well.
Ultimately,
this
means
that
on-call
rotations
are
healthier.
Engineers
are
happier
and
the
scope
of
our
daily
work
is
less
stressful.
B
Our
most
common
human
operation
is
replacing
a
node
instance
of
M
30
B.
Let's
see
what
that
looks
like
with
our
operator,
so
the
first
difference
that
we
noticed
in
our
operations
is
that,
whereas
we
previously
uniquely
identified
M
30
be
instances
by
hostname
now
we
also
have
to
take
into
account
the
storage
attached,
while
the
pod
names
stay
the
same
when
working
with
staple
sets.
B
A
Next
kubernetes
takes
care
of
scheduling
a
new
pod
for
us
with
new
storage
attached.
If
necessary,
our
operator
uses
our
regular
tooling
to
publish
a
new
desired
state
into
the
cluster
topology.
In
this
case,
we
would
like
the
pod
with
the
same
but
new
storage
attached
to
replace
the
pod
that
had
failed
next,
the
m3
DB
instances
begins
streaming
data
to
the
new
to
its
new
pier
once
the
cluster
is
healthy.
Again,
the
m3
DB
instances
update
the
current
state
in
the
topology
to
indicate
that
the
replace
has
finished.
A
Putting
m3
DB
on
kubernetes
taught
us
some
really
interesting
lessons.
The
first
it
was
that
showed
us
assumptions
we
had
made
in
building
m3
DB
for
a
human
operated
world
that
didn't
hold
true
on
kubernetes,
and
we
were
able
to
make
our
software
more
robust
because
of
it.
We
were
coming
from
a
world
where
one
host
ran
one
instance
of
m3
dB,
meaning
that
an
m3,
DB
instances,
identity
and
storage
was
equivalent
to
the
host
name
that
it
ran
on.
A
This
isn't
the
case
in
kubernetes,
where
the
pod
names
may
stay
the
same,
but
the
underlying
storage
can
change,
because
kubernetes
revealed
this
assumption
to
us.
We
were
able
to
make
em
3
DB
more
robust
by
enabling
an
instances
identity
to
take
into
account
more
factors
than
just
hosts
me
as
another
example.
We
discovered
a
race
condition
in
adding
hosts
to
clusters
that
we
had
never
encountered
before,
because
he
humans
simply
couldn't
do
the
steps
as
fast
as
kubernetes
could.
B
A
Over
the
course
of
building
our
operator,
we
think
that
some
of
the
lessons
we've
learned
might
be
helpful
to
you
if
you're
considering
kubernetes
for
your
database
or
otherwise
stateful
workload,
our
first
piece
of
advice
is
to
make
sure
that
your
workload
is
reliable
to
operate
outside
of
kubernetes
before
thinking
about
putting
it
on
kubernetes.
We
had
worked
really
hard
over
the
years
to
invest
in
tooling,
to
make
m3/d
be
easier
and
more
reliable
to
operate,
and
we
only
considered
automating
the
entire
end-to-end
lifecycle.
A
Our
second
piece
of
advice
is
to
attempt
to
model
your
stateful
workload
declaratively.
This
is
a
concept,
that's
really
core
to
kubernetes,
and
we
found
that
it
works
great
for
stateful
workloads
as
well.
We
were
already
doing
this
in
with
M
3
dB
we're
in
our
clustered
topologies
were
modeled
as
desired.
A
Additionally,
by
storing
our
topology
external
to
kubernetes,
we
avoided
a
hard
dependency
on
the
kubernetes
api
in
order
to
operate
our
clusters.
This
meant
that
eaton
disaster
scenarios,
even
if
the
kubernetes
api
was
fully
down,
we
could
still
make
cluster
topology
changes
so
long
as
the
instances
themselves
were
in.
B
Terms
of
project
planning,
it's
important
to
Ida
rate
on
each
step
on
the
staple
interaction.
Don't
try
to
mentally,
solve
everything
at
once
and
be
mindful
about
what
you
want
kubernetes
to
handle.
We
were
more
focused
on
reactive
stuff
to
start
with,
and,
ultimately
those
efforts
less
to
solve
proactive
hurdles
as
well.
It's
ok
for
some
things
to
still
be
left
to
humans.
Kubernetes
does
great
in
helping
bridge
gaps
along
the
way,
for
example,
in
our
use
case,
our
focus
where
the
operator
was
on
handling
day-to-day
operations.
B
We
would
definitely
still
need
human
intervention
in
some
edge
case
scenarios
such
as
recovery
if
we
somehow
lost
an
entire
region.
So
what
are
our
next
steps?
Our
operator
runs
smoothly
in
the
cloud
we're
scooping
out
the
work
as
to
how
we
run
kubernetes
in
our
internal
data.
Centers
we'd
also
like
to
explore
auto
scaling
of
clusters.
This
goes
hand-in-hand
with
capacity
planning
meetings.
Currently,
it's
pretty
easy
to
scale
up,
but
doing
so
automatically
would
be
another
milestone.
B
Also,
we
don't
yet
have
a
solution
for
managing
multiple
MDB
clusters
that
span
multiple
communities
clusters.
So
because
of
this,
we
don't
have
a
multi
region
solution,
yet
this
isn't
necessarily
a
blocker,
but
it
just
means
that
data
is
stuck
in
one
kubernetes
cluster
and
a
human
has
to
initiate
moving
it
to
another
cluster.
So
it's
definitely
something
that
needs
to
be
explored
with
deeper
planning.
A
Our
experience
putting
m3d
beyond
kubernetes
has
been
really
positive
so
far,
and
we're
excited
to
continue
our
kubernetes
journey.
We're
looking
at
incorporating
other
parts
of
the
m3
stack
into
our
operator,
such
as
our
aggregation
tier
and
our
query
engine,
but
we
started
with
m3
DB
as
it's.
The
trickiest
to
manage.
A
Kubernetes
has
enabled
us
to
take
on
new
use
cases
in
up
and
non-standard
environments
that
we
otherwise
would
have
had
to
say
no
to.
For
example,
we
now
have
production
use
cases
at
uber
where
we're
shipping
subsets
of
hardware
level
data
center
metrics
to
M
3
DB
clusters
running
on
kubernetes
in
the
cloud.
This
provides
us
external
observability
into
our
infrastructure
in
the
event
of
data
center
level
failures.
This
would
previously
have
been
too
difficult
to
stand
up
in
the
cloud
without
kubernetes.