►
Description
Running MySQL on Kubernetes [I] - Patrick Galbraith, Consultant
MySQL is the world's most popular open source database and there are a number of ways to run it on Kubernetes. This talk will cover each type of MySQL deployment strategy starting from a simple MySQL pod, to a asynchronous replicated master-slave, synchronous Galera cluster, and on to a Vitess clustering system which allows for horizontal scaling of MySQL and innately has built-in sharding, explaining how each is deployed, what features are available, and what type of application they lend themselves to.
About Patrick Galbraith
Patrick Galbraith has been involved in MySQL, Linux, and other Open Source (OSS) projects back to the early days of Slackware. He has worked broad spectrum of companies throughout his career, including Slashdot, MySQL, Blue Gecko, Hewlett-Packard, and landing recently at Dyn, an Oracle business. During his tenure in Hewlett-Packard's Advanced Technology Group (ATG) for Cloud and Open Source, Patrick had a focus on containerization with emphasis on Kubernetes; this focus, like all ATG endeavors, included participation in the associated OSS communities. Patrick has a keen, and long-standing interest in the effective marriage databases and Kubernetes with a bent toward MySQL. To this end, Patrick's "Galera Cluster for MySQL on Kubernetes" demonstration is a deep-rooted passion of his.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
Hi
everybody
thank
you
for
coming
to
my
talk.
I'm
extremely
nervous,
I'm
excited.
So
this
talk
is
about
MySQL
and
kubernetes,
and
my
name
is
Patrick
Galbraith.
Something
about
me.
I've
had
a
long
lasting
relationship
with
MySQL
I've
worked
on
MySQL
for
MySQL
various
other
companies
working
on
MySQL
I've
contributed
to
MySQL.
I
worked
on
my
various
MySQL
project's
memcache
UDF's
DVD
cue,
all
its
Perl
driver
I
still
am
m8
Co
maintainer
of
that
federated
storage.
A
Engine
I
contributed
the
original
example
of
running
Galera
on
kubernetes
in
the
examples
directory
for
kubernetes,
so
I
love,
learning
languages,
whether
it's
programming,
languages
or
spoken
languages,
I
love
being
with
my
family
outdoors,
I
love,
chili,
it's
the
best
place
I've
ever
been
and,
as
you
can
see
by
this
page,
I'm,
not
a
designer,
so
I
I
started
working
for
this.
This
great
company
Oracle
dine
in
September
and
I'm,
having
a
great
time
what
we
provide.
Most
of
you
probably
already
know.
This
is
DNS.
A
We
provide
DNS
for
some
of
the
world's
largest
and
most
admired
web
properties.
You
know
we
have
the
largest
fastest
and
most
resilient
DNS
networks
in
the
world.
Over
3,500
enterprises,
new
brands
like
netflix
twitter
linkedin,
we
also
provide
email
for
organizations
with
large
sending
needs.
We
also
are
building
a
cloud.
Oracle
cloud
infrastructure
and
Dyne
provides
edge
services
for
that
cloud
and
we're
hiring.
This
is
on
the
slide.
So
please
do
take
a
look.
A
So
what
is
this
presentation
about?
Well,
it's
about
running
MySQL
on
kubernetes
and
some
of
those
things
with
databases
are
extremely
complex.
Pieces
of
software
and
kubernetes
actually
makes
it
easier
to
run
databases
we're
going
to
talk
about
stateful
applications
which
database
is
a
complex
stateful
application,
and
in
that
we're
going
to
talk
about
stateful
sets
we're
gonna
showcase
this
with
Vitesse.
A
By
talking
about
the
test
and
doing
a
demo,
we're
also
going
to
talk
about
operators
and
I'm
going
to
be
very
excited
to
talk
about
the
MySQL
operator
that
Oracle
has
been
developing
and
give
a
demo
of
that
with
time
permitting
I
have
some
extra
slides,
I
went
crazy,
I
had
60
slides
at
one
point.
Somebody
said
this
is
a
two-day
seminar.
So
that's
why
I'm
speaking
you
know
with
some
cadence
here
and
please
do
save
questions
to
the
end.
A
So
it's
a
tale
of
two
open-source
projects
we
have
MySQL,
which
has
been
you
know,
and
software
year
has
been
around
for
twenty
years.
That's
quite
a
while,
and
it
certainly
is
ubiquitous
and
everywhere
and
used
every
so
many
places,
and
there
are
a
lot
of
variations
of
MySQL
permutations,
I
heard
even
heard.
Somebody
was
telling
me
about
the
work
that
Ali
Baba's
done.
I
know,
friends
at
Facebook,
they've
made
their
customization
and
then
there's
kubernetes
the
Linux
of
the
cloud.
A
As
we
heard
it
at
this
conference,
it's
I
got
to
say
the
fastest
growing
open-source
project.
It's
application
deployment
done
right.
Definitely,
community
driven,
look
at
all
the
people
here,
4000
people,
that's
fantastic
and
it's
a
moving
target
rapidly
developing
when
I
started
this
preparing
for
this
talk,
I
had
an
understanding
of
when
I
had
put
together.
The
original
Galera
example
and
things
have
changed
so
I
had
to
do
a
little
bit
of
catching
up
so
database
with
databases.
It's
a
question
of
pets
versus
cattle.
A
database
containers
tend
to
be
more
pets
than
cattle.
A
That
seems
obvious.
You
care
greatly
about
them
like
your
pet,
a
pod
of
multiple
databases
or
instances
or
one
a
you
know
you
the
description
of
pod.
Is
we
get
to
run
a
bunch
of
containers
on
one
node?
Well,
we
don't
necessarily
want
that
with
databases
and
the
documentation
that
says
with
four
pods,
it
says
kubernetes
pods
are
mortal
mortality
is
not
something
we
necessarily
like
to
see
as
an
attribute
in
databases.
A
database
is
a
complex,
stateful
application.
A
A
So
here's
some
ingredients
I
like
this
picture
of
shots,
I
love
shots,
they're
delicious,
and
so
we
have
some
delicious
ingredients
with
kubernetes.
We
have
services
which
I'm
not
going
to
talk
about
what
that
is.
Everybody
knows
what
that
as
persistent
volumes
per
citizen
volume,
claims
and
storage
classes,
and
then,
of
course,
we
have
a
newer
resource
on
the
block
stateful
sets
and
then,
within
that
there
we
can
talk
about
init
containers.
A
If
we
have
time
volume
claim
templates
and
then
there
there
are
operators
which
allow
you
to
extend
kubernetes
and
custom
with
the
utilize
custom
resource
definition,
then
we
have
sidecar
containers
to
do
some
handy
tasks
for
us
and
node
selector
I
was
going
to
talk
about
this.
I
probably
won't
have
time,
but
it's
definitely
something
to
know
about
for
databases
if
you
want
to
run
a
database
on
a
particular
node.
A
So,
first
of
all
we're
going
to
talk
about
stateful
sets
so
stateful
sets
provide
guarantees
about
ordering
and
uniqueness
of
pod
resources
and
pods
themselves
and
the
way
that
their
names
they
main
persist,
maintain
persistent
sticky
identity
for
each
of
the
products
rescheduling.
It
also
ensures
ordinality,
it's
good
when
it.
When
you
run
pods
within
a
stateful,
said
it's
going
to
be
zero
through
whatever
number
you're
going
to
run
and
they're
gonna
be
named
that
way,
they're
also
interchangeable,
and
they
won't.
You
won't
run.
It
run
a
pod
with
the
same
name.
A
You
won't
have
two
of
the
same
thing:
they
have
a
stable
network
identity.
First
of
all,
the
name
of
them
is
whatever
the
service
name
is
ordinal
of
how
many
over
ever
being
run,
and
then
you
create
a
headless
service.
If
we
have
time
we
can
discuss
this
and
the
that
is
the
domain
name
for
that
pod
and
that's
how
you
would
access
it.
They
have
stable
and
stable
storage
and
stateful
resources,
pods
remount,
the
same
storage
through
restarts.
So
when
there's
a
restart,
it's
the
same
thing
as
it
was
before
safety.
A
We
don't
scale
if
it's
unhealthy,
so
we
now
have
a
I'm
going
to
show
these
two
two
illustrations:
they're,
really
good
and
kind
of
bringing
the
point
home
as
to
what
the
differences
are.
So
with
a
deployment
replica
set
you
there
not
start
at
any
specific
order.
The
name
is
you.
You
have
a
name,
and
then
you
have
a
numeric
value
of
the
the
deployment
or
whatever
idea
there
is,
and
then
you
have
a
random
alphanumeric
and
that
changes
from
for
each
pod
they're
all
different.
A
It's
not
fun
to
try
to
log
and
figure
out
what
containers
are
running,
because
you
have
to
look
and
see
what
they're
called
and
it'll
still
scale
if
it,
if
there's
a
crash
and
it'll
replace
with
another
non
unique
name,
but
with
a
stateful
set,
what
we
get
we're
started
in
order,
you
don't
have
to
sit
there
and
say:
I
want
to
run
this
one
this
one
this
one
like
I
did
with
my
original
example.
The
names
are
unique
and
distinct,
and
it
must
replace.
A
If
there's
a
failed
note,
it's
going
to
replace
that
same
note,
and
it's
going
to
be
that
same
note.
For
all
intents
and
purposes,
and
you
don't
scale
the
cluster
when
it's
unhealthy,
so
then
we
have
operators,
it's
an
application-specific
domain
controller.
It
encodes
application-specific
domain
knowledge
through
extending
the
kubernetes
api.
What
does
this
mean?
This
means
we
as
operators.
We
do
certain
things.
You
know
if
I
run
the
Galera
cluster
I,
do
it
in
a
certain
order:
I
maintain
it
in
a
certain
way
with
operators,
you
build
that
into
it.
A
You
build
that
into
the
custom
resource
definition.
When
you
create
your
operator,
it
enables
users
to
create,
configure
and
manage
stateful
applications
that,
because
you
can
add
that
functionality
in
there
and
it
builds
upon
resources
and
controller
concepts
and
it
leverages
kubernetes
primitives
like
replica
sets,
stateful
sets
and
services
and
other
resources
an
operator
executes
common
application
tasks.
You
design
it
to
run
whatever
you
want
within
your
application
and
it
does
it
for
you
and
hides
all
the
gory
details.
A
It's
all
installed
as
a
deployment,
and
once
it's
running
you
utilize
it
and
you
run
it
with
custom
resource
definition
types.
Some
of
these
here
you
see
at
CD,
cluster,
Prometheus
and
MySQL
cluster
in
my
school
backup,
which
I'm
excited
to
talk
about
so
I
like
this
graphic.
This
is
when
I
copped
from
choruses
website,
where
they
discuss
operators,
but
it's
really
and
I
solved
this
morning
in
the
keynote.
There
was
something
just
like
this:
what
an
operator
does
is
it
observes
the
state
of
what
you
have
running
and
in
this
example
it
shows.
A
Oh
I
only
have
two
running:
oh
and
one
of
them
happens
to
be
three
point.
One
point:
one:
I
really
want
three
point,
two
point:
ten
and
that's
where
the
analysis
kicks
in
analyze,
and
it
says
it
should
be.
The
version
2.3
point
two
point
ten,
and
there
should
be
three
members
so
now
we're
gonna
act,
we're
gonna,
bring
up
that
missing
node
and
we're
gonna
also
upgrade.
A
So
this
graphic
here
it
was
more
about
me
wanting
to
talk
about
stateful
sets,
but
it
works
well
with
operators
as
well.
This
is
this
is
an
example:
Anthony
a
did
of
a
simple
MySQL
replication
and
what
it
utilizes
is.
You
have
a
simple
pod
with
two
init
containers.
The
first
one
is
in
at
MySQL.
It
sets
up
the
configuration
file
for
MySQL,
whether
it's
a
master
or
slave,
and
then
there's
clone
MySQL.
What
it
does
is
it
receives
a
backup
from
the
extra
backup.
A
If
it's
a
master,
it
doesn't
do
that
and
it
does
this
based
off
of
it
being
ordinal
number
zero,
and
then
there
are
the
app
containers
which
ql,
obviously
and
then
there's
an
extra
backup
container
extra
backup
and
what
it
does
is
once
mysql
is
up
and
running
as
it
starts
a
backup
so
that
the
next
subsequent
node
that's
running,
can
receive
the
back
up
through
the
init
container.
And
why
do
I
have
this
up
here?
A
Well,
there's
some
complexity
to
this,
and
this
is
the
kind
of
thing
that
you
would
put
into
your
operator
and
that's:
what's
really
cool
about
operators,
so
a
MySQL
operator
we
have
arrived
at
this
point.
I
always
wanted
to
do
this
as
one
of
these
things
that
I
thought
oh
I'd
be
great
to
do
this
well
I
happen
to
work
at
Oracle
now,
and
we
have
a
group
in
bristl
who's
developed
this
and
it's
scheduled
for
release
early
2018.
You
can
deploy
a
highly
available
clustered
ql
instance
unto
kubernetes
with
a
single
command.
A
So
MySQL
operator,
some
of
the
features
I've
already
mentioned
some
of
them,
but
I.
This
is
a
nice
graphic.
You
can
see
the
operator
here,
it's
doing
all
of
the
tasks
with
the
MySQL
cluster.
You
would
normally
do
as
an
operator
and
the
kubernetes
api
is
through
custom
resource
definitions.
Has
it
run
those
things
it
provides
configuration
for
your
mysql
cluster,
with
simple
kubernetes
objects,
like
secrets
and
config
maps
and
backups
you
can
define
what
kind
of
backups
you
want.
A
What
the
policies
are
to
make
sure
your
data
is
always
protected,
on-demand
or
scheduled,
as
I
mentioned.
Here's
an
example
of
trying
to
see
where
we
are
on
time,
so
here's
an
example
of
creating
a
cluster.
We
did,
we
create
a,
we
define
our
persistent
volume
and
it's
file
here,
and
then
we
have
the
MySQL
cluster
that
we
define.
We
say
we
want
three
of
them.
We
want
to
give
it
a
name
of
example:
MySQL
cluster,
with
volume
we
specify
M,
claim
template,
I
love
these.
A
These
are
great,
they
just
happen
to
work,
and
you
say
what
storage
class
name
you
want
and
you
submit
it
and
it
creates
a
cluster
and
then
with
a
backup,
same
simple
situation.
You
have
your
cluster,
we
build
test
cluster
and
then
you
specify
that's
what
cluster
you
want
to
backup.
You
say
what
your
credentials
are.
Oci
is
our
object,
storage,
you
know
just
like
s3
or,
and
you
specified
that
that's
what
you
want
to
backup
and
it
backs
it
up
and
the
commands
are.
A
You
know
you
use
cube
CTL
to
create
it,
both
your
backup
and
your
cluster.
You
can
also
do
cube.
Ctl
get
MySQL
backups
to
see
what
backups
you
have
and
then
you
can
get
a
specific
backup.
You
can
also
describe
back
if
I
don't
have
that
in
there,
but
you'll
see
that
in
the
example.
So
now
we're
going
to
run
a
demo
drumroll.
A
A
We
see
that
it's
running
replica
set.
So
at
this
point
we
can
now
you
utilize
the
operator,
we're
gonna,
tell
the
logs
of
the
operator
and
see
what's
going
on,
it's
kind
of
I
like
the
way.
This
was
recorded
that
you
can
see
this
now
we're
going
to
go
into
the
cluster
manifest
file,
we're
going
to
do
three
replicas,
we're
gonna
call
it
MySQL
test.
Cluster
cube,
CTL,
create
dash
F,
the
name
of
that
and
at
that
point
you
can
see
the
logs
that
it's
starting
to
bring
up
the
cluster.
A
A
A
A
We're
gonna
use
the
MySQL
I,
know
DB
cluster
made
a
data
schema
and
then
we're
gonna
find
out
what
the
group
membership
is
from
hosts
and
there
we
have
a
three
node
cluster.
That
was
easy,
wasn't
it
so?
The
next
thing
I'm
going
to
show
you
is
a
backup.
This
is
even
easier.
So
initially
we
have
nothing
in
our
object
store.
A
A
A
So
now
we're
gonna
talk
about
the
test.
The
test
is
charting
on
a
silver
platter.
I
know
it's
not
a
silver
platter,
but
I
love.
The
album
I
doesn't
have
anything
to
do
with
my
age.
So
it's
a
database
solution
for
deploying
and
scaling
and
managing
large
clusters
of
MySQL
instances.
It
has
built-in
sharding.
It's
been
YouTube's
database
store
since
2011.
A
It's
what
slack
uses
it
looks
like
a
MySQL
database
in
terms
of
usage,
but
underneath
it
can
be
just
sharted
in
any
way
that
you
want
it
to
be.
It's
cloud
ready.
The
cloud
is
coming
is
sugu
would
say
it
helps
you
scale
with
transparent,
dynamic,
sharding
and
the
ability
to
continue
charting
in
or
out
depending
on
what
you
want
to
do.
It
also
has
cluster
management
tools
for
backups
shards
and
schema
management.
A
It
talks,
MySQL,
client
protocol,
G,
RPC
and
G
RPC.
It
provides
connection
pooling
these
are
go
connection,
pool
that
talks
to
a
small
MySQL
connection,
pool
it
protects
MySQL
with
MySQL,
with
query
deduping,
rewriting
sanitization.
Putting
limits
on
unbounded
causes
and
a
number
of
other
things,
there
are
monitoring
tools
that
are
built
into
all
the
the
test.
Binaries.
A
So
here
are
the
components
of
the
test
we
have.
First
of
all,
we,
the
theory,
you
talk.
Your
application
talks
to
VT
gate
is
a
lightweight
proxy
used
for
routing
queries
to
the
appropriate
VT
tablet
and
it
also
reassembles
the
result
sets
from
the
different
shards,
so
VT
gate
then
talks
a
V,
T
tablet.
Vt
tablet
pause
are
comprised
of
VT
tablet
itself,
which
itself
is
a
proxy
in
mysql.
Vt
tablet
is
a
proxy
that
does
all
of
that
query.
A
Sanitization
and
you
know
blacklisting
deduping
putting
but
putting
a
limit
on
unbounded
clock
where
it
clauses.
Then
next
we
have
the
we
have
the
topology
server.
This
topology
server
is
essentially
a
service
all
ends
in
front
of
SED
and
all
of
the
data.
The
coordination
data,
the
key
space,
keep
key
space
graph
used
to
determine
where
the
shards
which
shards
use
use.
It's
also
used
for
initial
discovery
of.
What's
there
and
then
you
have
VT
c
TLD
that
provides
a
web
interface
and
it
does
all
the
cluster
management
work.
A
And
then
another
piece
in
here
that
I
added
that's
not
on
the
graphic
on
the
website
is
Orkut.
Orchestrator
Orchestrator
pertains
to
repairing
and
it
talks
a
VT
c
TLD
and
that's
what
it
through
that
it
does
the
repair
in
ting
when,
when
that
needs
to
occur,
and
then
there's
V
T
CTL,
that's
the
command
line
for
interacting
with
the
tests
through
the
through
the
topology
server.
A
So
charting
a
shard
consists
of
a
set,
as
I
mentioned,
a
single
master
that
handles
writes
and
one
or
more
slaves
a
handle
reads:
you
can
split
horizontally.
You
can
splitter
merge
shards
in
a
sharded
key
space.
You
can
also
do
vertical
sharding
moving
tables
from
an
uncharted
key
space
to
a
different
key
space.
You
can
do
restarting
split,
merge,
add
new
cells
and
replicas.
It
does
this
through
filtered
replication
using
GT
ID,
which
is
the
key
ingredient
to
ensuring
that
your
data
goes
where
you
expect
to
go.
A
There's
also
the
concept
of
this
V
schema.
This
is
a
Vitesse
schema.
This
is
essentially
the
schema
for
how
you
shard
think
of
it
that
that
way,
it
ties
together.
All
the
databases
simply
put
it
contains
information
needed
to
make
the
tests
look
like
a
single
database
server
and
then
there's
V
index.
This
is
a
cross
shard
index
that
provides
a
way
to
map
what
in
your
table
what
it
would
call
them
to
your
key
space.
Id
and
I'll
show
you
an
example
of
this.
So
right
here
we
have,
you
know
on
the
Left.
A
A
Then
we'd
say
what
tables
the
user
and
we
say
what
column
we're
going
to
shard
by
and
what
the
V
index
name
is,
and
it
happens
to
be
hash,
as
we
just
saw
up
there,
and
this
is
what
you
submit
to
Vitesse
and
that's
how
the
test
knows
how
to
shard
your
data,
there's
also
the
concept
of
repairing
we
can
change
or
shards
master
tablet
to
another
host.
We
can
change
a
slave
tablet
to
a
different
master,
as
I
mentioned,
GT
IDs
the
magic
in
all
of
this.
A
A
You
can
look
at
in
your
spare
time
when
you
look
at
my
slides.
So
now
we
get
to
the
fun
part.
We're
gonna
do
a
demo
and
what
we're
gonna
do
here
is
we're
going
to
create
two
shards
we're
gonna
load,
a
database
schema
and
V
schema
scale.
One
shard
from
two
replicas
to
three,
then
we're
gonna
delete
the
stateful
set,
we're
gonna,
pull
the
carpet
from
the
rug
from
under
its
feet
and
then
delete
the
master
pod
and
then
see
the
tests.
Do
some
magic
of
repair
in
ting.
This
demo
is
made
possible.
A
A
There
should
be
nothing
here
with
Vth
status
or
vtc
TLD
helm
chart
is
installed
so
once
this
is
installed,
we
should
start
being
able
to
look
at
our
stateful
set
and
then
see
what
pods
are
coming
up
the
zone.
One
main
those
are
the
two
shards
should
watch
them.
Come
up,
there'll
be
some
errors.
This
is
a
bug.
I
won't
go
into
this.
This
is
a
known
issue.
It's
nothing
to
worry
about.
A
A
So
now
you'll
see
replication
get
started
and
a
master
in
a
replica
setup.
You
can
see
here.
It's
already
happened
with
the
ad
shard
and
now,
with
the
other
shard
it's
coming
up
as
well.
Both
shards
should
be
ready
at
this
point.
We
won't
see
any
traffic
on
it,
yet
we're
not
running
anything
against
it.
That's
going
to
change
soon,
no
traffic,
it's
nice
to
have
graphs
there.
You
won't
see
anything
in
the
query
log
either.
There's
a
nice
query!
A
Log
UI
there
that's
built
in
to
vtc
BTC
ltd,
TLD
I
get
that
messed
up
on
the
tongue-twister
here.
So
now
we're
reloading
the
script
is
loaded.
We
now
have
a
master
in
a
replica,
we're
doing
a
backup
right
now
and
it's
not
serving
while
we
do
that
and
watch
the
logs
a
little
bit
to
see.
What's
going
on.
A
A
A
A
Also
gonna,
look
that
you
know
we
never
did
do
this
when
we
were
initializing
it,
but
we're
gonna.
Look
at
that.
The
schema
is
actually
there
we're
gonna
look
at
the
query.
Log
query:
log
UI,
you
see
all
the
different
queries
running
against
the
the
master
at
this
point.
All
right
queries,
write
statements,
I
mean.
A
A
A
A
A
So
that's
the
end
of
the
Vitesse
demo.
What
time
is
it
421?
So
we
have
about
what
is
it
about
four
minutes
for
questions?
I?
You
know
I
I
have
these
extra
slides,
please?
When
you
look
at
these
slides,
please
look
at
collect
beats.
This
is
a
great
thing.
I
wanted
to
talk
about
was
a
friend
Vijay
Samuel
who
works
for
eBay
who's
written.
This
I
wanted
to
cover
it,
because
I
wanted
to
talk
more
about
monitoring
and
I
also
have
some
other
MySQL
installation
examples,
because
I
wanted
to
do
simple.