►
From YouTube: Building a Database as a Service on Kubernetes - Abhi Vaidyanatha & Lucy Burns, PlanetScale
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Building a Database as a Service on Kubernetes - Abhi Vaidyanatha & Lucy Burns, PlanetScale
Stateful, scalable storage on Kubernetes is an unsolved problem. Creating it as a service is even more difficult. The cloud-native ecosystem offers many tools such as the operator-sdk, Prometheus, Grafana, etcd, Vitess, and much more, but integrating them isn't necessarily intuitive. Two of PlanetScale's employees that have engineered and managed the project describe the journey of leveraging all of these open source technologies to build out a database as a service on Kubernetes.
https://sched.co/Uabw
A
B
B
A
B
Cool
so
we're
gonna
have
our
nice
little
minimalistic
agenda
for
us
today.
First
we're
going
to
find
the
problem.
As
we
said,
stateful
workloads
in
a
stateless
environment
is
very
difficult
offering
the
service
is
even
more
difficult,
so
we're
gonna
kind
of
go
through
the
historical
journey
of
how
we
got
there
and
then
go
into
how
we
attack
the
problem.
Yeah.
B
So,
if
you're
able
to
catch
the
awesome
talk
by
slack's
engineers,
Raphael
and
Guido
you're
able
to
kind
of
get
into
this
idea
of
a
mutable
infrastructure.
So
before
we
had
a
mutable
it
structure,
we
had
these
wonderful,
like
spicy
deployments
that
you
really
didn't
know
or
what
were
what
was
being
deployed
in
production
and
stuff
was
really
what
was
not
deterministic
and
needed
to
be
updated
and
modified
in
place.
Now
we
know
that
it
has
a
small
scale
manually.
B
Updating
and
modifying
your
infrastructure
in
place
is
not
that
hard,
but
as
it
grows
out,
it
becomes
very
difficult.
However,
we
ran
into
one
issue.
The
issue
is
that
redeploying
infrastructure
is
expensive,
so
as
cloud
providers
kind
of
figured
out
how
to
make
redeploying
infrastructure,
they
automated
be
quick
and
to
figure
out
persistence
with
with
new
and
old
infrastructure.
These
problems
started:
go
away
and
containers
solved
the
issue
of
being
of
new
deployments
being
deterministic
and
kubernetes
solved
the
issue
of
immutable
infrastructure
being
able
to
scale
so.
A
Now
we're
in
this
place,
where
we
have
our
applications
and
kubernetes
receiving
all
the
benefits
that
are
be
described,
and
now
we
come
to
the
question
of
what
do
I
do
with
my
databases
and
at
first
this
question
might
be
feel
little
antithetical.
Do
I
really
want
to
run
stateful
workloads
in
a
stateless
environment
and
in
fact,
most
experts
recommend
that
you
don't
do
that
and
that's
because
you
can't
take
the
longevity
of
your
pod
in
which
your
master
is
running
for
granted.
A
A
B
As
with
any
hard
problem
comes
a
very
strict
form
of
a
strict
outline
for
what
we
want
from
our
product,
so
we're
going
to
define
what
it
means
to
be
a
cloud
native,
because
we
feel
that
a
cloud
native
database
has
all
of
these
really
important
qualities
and
you're
able
to
trust
a
cloud
native
database
with
your
with
your
data.
The
first
thing,
and
arguably
the
most
important
thing-
is
reliability.
Fault
tolerance
is
incredibly
important
and
whether
you're
losing
a
pod,
a
node,
a
replica,
a
master,
a
data
center.
B
A
Guys
have
all
managed
databases,
it
sounds
like
so
you
know
it
is
certain
point:
you'll
hit
the
question
of
scale
so
whether
you
start
small
and
grow
big
and
want
to
scale
out
or
if
your
application
sees
traffic,
spikes
from
events
like
Black
Friday
or
a
Padres
game
or
some
other
event.
You
want
to
be
able
to
scale
up
and
scale
down
as
you
need.
A
Additionally,
we
want
to
be
able
to
treat
our
database
system
as
one
entity,
even
though
there
may
be
moving
parts
behind
it.
We
want
something.
That's
logically
singular
in
place
of
some
instance.
What
we
don't
want
is
multiple
levels
of
architecture,
kind
of
duct-taped,
together
with
custom
code.
We
want
one
entity
so.
B
Then
we
get
to
aunty
horticulture
ISM,
so
you
guys
are
all
familiar
with
the
pets
versus
cattle
analogy.
The
fact
that
you
don't
want
to
be
managing
and
tending
to
pets,
and
you
want
to
be
able
to
treat
your
infrastructure
as
cattle,
it's
2019.
So
we
have
a
plant-based
analogy
now
we
want
to
use
the
analogy
versus
gardens
and
row
crops.
So
if
you
think
about
a
garden,
you
have
your
little
tomato
plant,
you
care
about
that
tomato
plant.
Like
you,
you
tend
to
it.
You
care
for
its
life.
B
A
So,
of
course,
the
product
manager
in
me
was
like
great
we've
defined
what
we
want.
How
do
we
build
the
Minimum
Viable
Product?
How
do
we
scale
this
project
down
to
the
bare
necessity
and
what
we
realized
really
quickly
is
a
database
is
such
a
fundamental
part
of
your
infrastructure
that
you
can't
really
scale
it
down
not
easily.
You
do
want
to
have
monitoring
and
alerts
and
scaling,
and
all
of
these
other
things
there's
actually
a
really
big
surface
area
for
just
creating
a
standard
database.
B
The
way
I
like
to
describe
it
is,
if
we
have
our,
if
we
rate
our
database
as
a
service
product
and
from
one
being
up
like
enterprise
piping
dev
null
and
is
a
fully
functioning
cloud
native
databases
service.
We
have
to
release
at
a
10
and
then
develop.
Quality
of
life
features
to
a
15.
So
we
decided
that
a
no
compromise
product
was
absolutely
the
way
to
go.
A
So
how
are
we
gonna
get
there?
We
have
this
huge
task
ahead
of
us:
a
planet
scale.
We
decided
to
go
all-in
on
open
source
technology
for
a
few
different
reasons.
First
of
all,
some
of
the
technology
we
were
looking
at
that
was
open.
Source
had
been
developed
for
the
container
ecosystem,
which
made
it
a
perfect
fit
for
what
we
were
trying
to
create.
Additionally,
we
knew
the
software
had
been
created
by
motivated
contributors,
contained
bleeding
edge
technology
and
was
of
a
very
high
quality,
and
we
wanted
to
use
that.
A
B
We
recognize
that
there
are
some
drawbacks
to
open
source
and,
if
you,
if
you
want
a
specific
feature
in
your
favorite,
open
source
products
to
be
to
exist,
you
have
to
spend
your
own
developers
time
going
and
writing
that
feature.
Now.
The
thing
is,
as
Lucy
said,
we
actually
view
this
as
an
opportunity.
We
love
engaging
with
the
open
source
community
and
we
feel
that,
from
contributing
back
and
from
helping
out
the
communities,
we
can
actually
create
more
support
and
kind
of
pick
out.
A
So
the
test
lies
at
the
heart
of
our
database
as
a
service.
In
fact,
planet-scale
was
created
to
provide
a
larger
audience
for
Vitesse.
For
those
of
you
who
aren't
familiar
with
the
test,
it
was
created
in
2010
at
YouTube
to
help
YouTube
through
scale
through
its
hyper
growth
period.
Today
it
sees
production
traffic
from
companies
like
slack
square
JD,
comm
and
many
more
Vitesse
is
a
great
database
management
system
because
it
offers
observability
monitoring
permissions
and
you
can
scale
horizontally.
A
A
A
So,
ok,
we
like
the
tests,
but
the
tests
can
be
a
really
big
challenge
to
configure
and
what
we
wanted
to
do
was
to
take
our
in-house
expertise
and
create
an
opinionated
package
that
people
could
use
out
of
the
box.
This
naturally
led
us
to
a
database
as
a
service.
Once
we
decided
that
ok,
we're
gonna
have
a
database
as
a
service
for
virtus.
We
ran
into
our
first
challenge.
The
test
is
designed
to
manage
one
large
distributed
system
with
shared
infrastructure.
A
A
B
B
This
is
really
cool,
because
a
lot
of
them
exist
in
open
source
for
all
your
favorite
open
source
products,
but
there
isn't
a
there,
isn't
a
really
solid
one
for
the
test,
so
we
had
to
write
one
ourselves.
We
spent
a
lot
of
time
writing
our
first
version
over
the
test
operator,
which
spun
up
all
of
our
different
database
components,
but
we
did
it
on
a
really
old
version
of
the
operator
SDK
and
while
the
operator
SK
is
fantastic,
we
it
it
had
like
progressed
a
lot.
B
Oh,
it
progressed
a
lot
since
then
and
had
breaking
changes
that
we
were
kind
of
stuck
to.
So
we
made
the
choice
to
completely
rewrite
our
operator,
and
that
was
probably
one
of
the
best
choices
we
ever
made
because
the
new
operator,
the
newer
versions
of
the
operator
SDK,
have
the
new
controller
runtime,
which
basically
improved
our
deployment
speed
by
like
80
percent,
and
it
has
access
to
auto-generated
documentation.
So
definitely
take
advantage
that,
if
you're
planning
on
developing
developing
your
own
operator,
it
is
absolutely
fantastic.
Now.
B
That
being
said,
we
couldn't
use
operators
for
everything,
while
the
SCD
operator
is
really
good.
It
has
a
lot
of
really
cool
features,
such
as
a
recovery
and
auto
scaling.
We
were
finding
that
it
was
not
actually
possible
for
us
to
recover
from
Chora
from
a
loss
of
quorum
unless
we
took
backups
all
the
time
now.
The
thing
is
is
that
at
CD
with
Vitesse
serves
one
purpose,
it
has
one
contract
and
that's
to
essentially
tell
where
everything
is
restore
our
sharding
scheme.
B
We
store
like
how
we
like
how
you're,
starting
where
your,
where
your
databases
are
and
what
you're
serving
graph
looks
like
in
SA
D.
We
rely
on
consistency
and
not
necessarily
for
with
performance
for
this,
so
we
actually
don't
necessarily
need
high
availability
on
this.
We
simply
just
we
simply
just
need
it
to
work.
B
So
what
we
did
is,
instead
of
using
that
C
D
operator,
we
used
our
of
the
test
operator
to
bring
up
at
C
D
clusters
as
staple
sets,
which,
with
each
database
that
we
provisioned
and
we
backed
all
of
those
with
the
persistent
volumes.
So
we
actually
didn't
have
to
worry
about
taking
backups
for
our
XE
d
instances
anymore
additionally
ma
during
school,
you
need
it
for
any
database
product.
Thankfully,
we
have
the
de
facto
cloud
native
monitoring
available
to
us
through
open
source.
B
Prometheus's
awesome
is
well
accepted
in
many
different
up
in
many
different
pieces
of
software.
Prometheus
is
so
through
lots
of
different
pieces
of
piece
of
software.
It
is
natively
integrated
by
the
with
the
slash,
metrics
and
endpoint,
or
a
lot
of
companies
are
or
pieces
of
software
run
it
as
a
sidecar
process.
So
because,
because
it's
so
well
known,
it
was
kind
of
like
a
natural
progression
for
us
to
use
it.
Additionally,
Prometheus
is
really
nice
in
terms
of
the
UI.
B
It's
very
clear
what
your
configuration
is
as
someone
who
wants
to
know
exactly
how
like
how
my
infrastructure
is
is
configured.
If
you
just
go
straight
into
the
Prometheus,
you
I
click
on
config.
It
tells
you
exactly
what
services
you're
trying
to
discover
how
many
targets
you
have
and
whether
you're
actually
discovering
them.
So
it's
really
it's
really
convenient.
It
looks
beautiful
and,
finally,
the
kind
of
one
of
the
interesting
challenges
that
we
had
with
with
prometheus
was
actually
configuration.
B
The
Prometheus
operator
works
fantastic
and
that's
actually
how
we
deploy
Prometheus
when
you,
when
we
specify
monitoring
in
our
in
our
top-level
CRD,
we
talk
to
the
Prometheus
operator,
brings
up
prometheus
pods
for
us.
However,
there's
a
lot
of
there's
a
lot
of
configuration
that
needs
to
go
with
it
and
we
kind
of
just
went
around
Google
and
Stack
Overflow
and
kind
of
just
put
together
this
like
configuration
Frankenstein,
which
looked
really
messy.
B
But
then
we
realized
that,
if
you
go
to,
if
you
go
to
the
core
OS
github
repository,
there's
something
called
cube
Prometheus,
which
actually
allows
you
to
audit
generate
all
of
that
configuration
for
you.
So
don't
don't
do
what
we
did
go
and
get
that
get
that
manifest
with
pin
dependencies
for
you.
So.
B
As
you're
kind
of
figuring
out,
our
the
test
operator
is
the
lifeblood
for
for
our
product.
It
spins
on
everything,
including
our
graph,
on
a
pod
which
immediately
pulls
in
the
data
source
from
Prometheus.
Now,
when
how
do
we
expose
that
graph
on
a
dashboard
to
the
user,
so
we
have
a
graph
on
a
link
in
our
database
as
a
service.
This
request,
when
you
click
on
that
link,
is
then
sent
to
a
level
seven
application
load
balancer.
Then
we
send
it
to
our
thin
occation
proxy.
B
B
Here's
our
wonderful
architecture,
diagram,
as
you
can
see,
the
plan
of
scale
operator
is
kind
of
the
the
core
of
it
that
spins
out
all
the
all
the
stuff
that
we
need.
When
you
get
an
API
request,
it
comes
in
the
form
of
a
planet
skill
cluster
CRD,
like
we
said
when
we're
building
out
a
cloud
native
database.
We
want
our
things
to
be
logically,
singular
everything
lives
under
this
plan
scale.
Cluster
CRD
play
the
plane
is
kala.
B
Cluster
CRD
changed
for
the
any,
whether
it's
a
create
update,
delete
is
sent
to
kubernetes
with
which
then
passes
it
over
to
the
plant
scale
operator,
which
is
in
charge
of
reconciling
that
planet
scale.
Cluster,
that
planet
skill
cluster,
then
spins
up
all
of
the
required
components,
it'll
spin
up
all
of
the
all
of
the
tests
and
its
sub
components.
B
If
you
want
to
learn
more
about
that,
we
have
lots
of
talks
about
the
tests
we
we
then
spin
up
our
Prometheus
pod,
which
then
is
autumn,
and
then
we
make
sure
to
spin
up
fauna
after
that,
which
then
captures
the
Prometheus
data
source.
And
then
we
spin
up
our
authentication
proxy,
which
which
allows
us
to
link
to
grow
fauna
and
to
other
to
our
stats
pages
that
are
that
calm
package
with
the
tests.
B
A
And
you
can
try
this
out
yourself
if
you
would
like
the
URL
is
console
dot,
planet-scale
comm.
So
what
I'm
going
to
go
ahead
and
do
is
create
a
cluster
when
I
do
this?
I
am
spinning
up
all
of
the
Vitesse
architecture
that
we
talked
about.
I
can
also
choose
the
platform
and
the
region
that
I
would
like
to
deploy
in
I'll
choose
AWS
u.s.
u.s.
East.
A
A
We
can
see
it's
a
single
region
deployment
here.
I
can
add
additional
resources,
whether
it's
CPU,
RAM
or
storage,
I
can
decide
on
the
number
of
instances
for
high
availability,
I
always
like
to
have
three
and
then
here's
the
really
fun
part
that
the
Tash
offers
I
can
go
ahead
and
deploy
an
uncharted
database.
So
it's
just
a
my
sequel
instance
or
I
can
start
out
sharded
and
create
a
sharding
scheme
that
will
route
my
queries
to
the
correct
shard
and
gain
scale
through
doing
that.
A
It
generally
takes
about
two
to
four
minutes
to
deploy
our
debate
to
deploy
our
database
depending
on
how
many
replicas
you're
using
so
what
I've
done
is
created
another
one
that
we
can
take
a
look
at,
so
we
don't
have
to
waste
our
time
on
that.
So
here
you
have
a
deploy
database.
You
can
see
how
long
it's
been
running.
A
B
All
right
so
was
that
hard
yeah
yeah
it
was
it
was.
It
was
pretty
hard,
but
the
thing
is
that
not
for
the
reasons
that
we
thought
it
was,
we
thought
it
would
be
hard
to
create
a
database
as
a
service,
because
we'd
have
to
create
all
the
internals
and
create
all
the
like
kind
of
the
lifeblood
of
the
prod
of
the
product.
But
the
real
answer
was:
it
was
a
lot
of
meta
infrastructure
that
we
had
to
build
around
there.
B
We
had
to
basically
provide
a
reliable
and
trust,
fit
and
trustworthy
database
to
solve
a
problem
that
a
lot
of
people
would
consider
has
never
been
solved
before
we
have
to
be
earn
an
environment
where
stuff
is
expected
to
go
down,
we
have
to
persist
or
persist.
Your
data
serve
traffic
in
a
very
dangerous
environment,
and
we
understand
that
of
network
boundaries
are
hard.
Our
back
is
hard.
Reconciliation
is
hard.
B
Consistency
is
hard
and
just
overall
configuration
is
kind
of
a
difficult
thing,
but
are
basically
the
main
functionalities
and
all
of
the
and
all
like
the
really
cool
things
that
that
that
we
use
to
serve
our
product
isn't
hard
because
a
lot,
because
we
have
so
many
really
cool
open-source
offerings.
And
it's
honestly
really
cool
that.
B
A
B
Like
I
said,
this
is
a
problem
that
a
lot
of
people
consider
is
not
salt,
yet
we
want
to
show
people
with
I
guess
through
our
database
as
a
service
or
just
through
the
open
source
to
test
lack
through
the
community.
Show
people
that
you
can
do
this,
and
we
want
to
talk
to
be
able
to
help
and
kind
of.
Angela
is
the
fact
that,
like
that,
it
is
very
possible.
The
test
was
built
to
do
this
and
that
this
problem
is
soon
gonna
become
thing
of
the
past.
On.
A
The
product
side,
we
want
to
offer
something
that
we've
been
calling
B
yok
or
bring
your
own
kubernetes
so
that
you
can
take
our
database-as-a-service
software
and
hook
directly
into
your
kubernetes
deployment
and
deploy
your
databases.
There
really
excited
about
that.
Additionally,
we
want
to
build
out
the
functionality
for
a
true
multi-cloud
cluster,
and
what
I
mean
by
that
is
I
want
to
be
able
to
spin
up
a
master
in
AWS
and
spin
up
replicas
and
GCC
and
have
the
master
and
replicas
communicate
across
cloud
platforms.
B
Currently,
the
way
that
horizontal
skill
works
it
it's
kind
of
it
is
a
manual
process,
but
it's
something
that
we
can
absolutely
automate.
The
time
will
come
where
we
can,
where
we
will
be
able
to
use
Vitesse
performance
characteristics
to
automatically
shard
your
database
for
you
and
to
keep
up
with
your
with
your
ever
growing
company
cool.
A
C
A
C
D
B
The
sub
level
objects
which
are
which
are
it
actually
creates
objects,
called
the
test,
cells
and
Vitesse
clusters
and
the
test
key
spaces,
and
we
actually
so
a
good
question,
because
we
actually
just
spent
a
lot
of
time
open
sourcing,
the
main
main
components
of
the
operator
and
very
shortly.
Everyone
will
be
able
to
use
our
open
source
to
test
operator,
to
deploy,
to
deploy
the
test
clusters
and
kubernetes,
and
so
and
everything
in
the
serving
path
will
be
completely
open-source
I.
B
A
D
C
C
So
each
company
can
have
multiple
clusterings
inside
them,
but
company
is
a
high-level
object
that
is
allocated
to
every
tenant
and
I'm,
trying
to
remember
whether
we
put
each
company
in
a
different
namespace
or
not,
but
the
intent
is
to
put
each
company
in
a
different
namespace
and
have
a
net
city
cluster
associated
with
each
company,
in
which
multiple
witness
clusters
that
that
company
creates
can
live.
That's
how
we
are
handling
multi-tenancy.
So
it's
the
same
overall
kubernetes
clusters
in
all
the
data
is
living,
but
it's
different
namespaces.
C
F
A
F
C
C
We
call
durability
using
disk,
but
you
can
also
run
the
tests
in
the
mode
that
slack
runs
it
that
we
call
as
Network
durability
and
in
that
mode
there
is
one
master,
two
replicas
and
we
configure
my
sequel
to
run
with
a
semi
sync
replication
turned
on,
which
means
that
the
master
does
not
acknowledge
a
particular
commit
as
having
succeeded
until
at
least
one
of
the
replicas.
All
the
data
associated
with
that
commit
in
its
relay
logs
right.
C
So
that's
how
we
and
we
use
anti
affinity
to
make
sure
that
your
master,
pod
and
replica
parts
are
not
on
the
same
host
the.
Thus,
if
the
master
part
goes
away,
we
choose
one
of
the
replicas
as
the
new
master,
which
has
all
the
data
for
committed
transactions
and
that's
how
you
get
Network
durability.
That
has
some
interesting
characteristics
because
we
don't
need
to
flush
to
disk
after
every
commit,
so
you're
right
throughput
actually
goes
up.
C
If
you
are
running
in
that
fashion
and
if
your
master
goes
down,
we
just
spin
up
a
new
replica
which
instantiates
itself
from
a
backup
we
say
with
GT
ID
with
every
backup
and
then
it
starts
replicating
as
of
that
GT
from
the
master
and
once
it
catches
up,
it
joins
the
serving
quorum.
So
that's
the
second
type
of
durability
and
slack
runs
like
that.
Currently,
the
database
or
service
doesn't
support
that,
but
that's
on
the
roadmap
also
any
other
questions.
G
B
B
We
assign
them
at
JWT,
and
so
what
what
we
do
is
we
we
use
the
kubernetes
api
to
go
and
to
go
and
look
at
their
at
whether
they
have
whether
they
have
the
what
they
provision,
what
and
whether
they
have
the
correct
permissions
for
it
and
what
we
see
if
that
matches
up
with
the
claims
in
their
JWT,
so
so
that
that's
just
that's
what
that's
mostly
what's
going
on
there
and
I.
Don't
know
if
that
answers
your
question
or
not
or.