►
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Democratizing MySQL: From Cloud Managed to Kubernetes Managed - Sachin Manpathak, Platform9 & Flavius Mecea, Presslabs SRL
Platform9, like many startups used public cloud based MySQL service as backend to quickly get started. But as their customer base grew, the public cloud costs became significant. Their goal for this project was ambitious - they wanted to run their own MySQL as a service without any in-house MySQL expertise. They also wished the model to scale well, achievable with a small team. Learn how Platform9 has reached their goal by using a highly flexible open-source MySQL Operator, written by Presslabs—a startup that aims at democratizing the WordPress hosting infrastructure by using Kubernetes and other open technologies. This is a back-to-back combo talk featuring 2 people who know the MySQL Operator better than anyone, but differently: the original builder & its heaviest user.
https://sched.co/MPYa
A
Hello,
everyone
I'm
happy
to
be
here
with
you
and
thank
you
for
joining
our
session
about
democratizing
my
sequel
cloud
managed
to
Burnett
is
managed.
My
name
is
Flavius
major
and
I'm
going
to
talk
about
the
story
of
building
press
loves
operator.
For
my
sequel,
this
is
a
joint
session
with
a
platform
9
searching
month
attack.
A
First
of
all,
I
want
to
start
representing
my
Mexico
operator
and
I
will
go
through
these
five
main
topics,
the
context
in
which
the
operator
was
born,
the
needs
that
we
had
in
mind
also
what
have
changed
and
what
we
achieved
also
some
challenges
that
we've
encountered
during
development
and,
finally,
the
project
status
and
some
future
plans.
First,
let
me
introduce
the
company
that
I
work
for
and
the
context
in
which
the
operator
was
born.
A
We
are
a
managed
wordpress
hosting
company,
doing
business
for
more
than
10
years.
We
started
as
a
WordPress
development
agency,
then
we
pivoted
towards
the
hosting
business
after
serving
both
publisher
and
enterprise
clients.
For
several
years,
we
came
to
realize
that
all
companies
in
the
global
top
tire
Enterprise
were
doing
the
same
thing,
including
us.
So
that's
when
we
started
thinking
about
open
source
infrastructure
that
could
become
the
WordPress
hosting
standard
name
rest
of
stack.
So
it's
not
just
about
that
stack.
A
A
A
Our
core
services
already
runs
on
kubernetes,
and
it
provides
a
lot
of
support
for
a
lot
of
integrations
like
search
management,
nginx
Prometheus
for
monitoring
and
so
on.
So
as
part
of
the
stack
we
needed
a
way
to
automate
certain
operations,
such
as
deploying
scaling,
maintaining
and
backups
for
my
sequel
for
that,
we've
identified
several
key
requirements
to
focus
on.
First,
we
wanted
something
that
it's
easy
to
operate,
something
that
doesn't
get
in
your
way.
A
Second,
we
need
an
elastic
service
to
help
us
scale
with
the
demand,
and
you
all
know
that
in
hosting
service,
uptime
is
paramount.
So
we
had
to
maximize
the
availability
for
service.
Also,
no
one
wants
to
lose
data,
especially
when
it
comes
to
someone
else's
data
and
in
order
to
reliably
operate
the
service
our
system.
We
need
a
method
to
observe,
what's
happening
from
the
top
down
to
the
request
level.
A
A
So
as
great
engineers
do,
we've
ended
up
building
a
solution.
Ourself
a
kubernetes
operator
for
managing
magical
cluster
with
asynchronous
or
semi
synchronous
replication.
So
in
the
past
ten
years
we've
identified
several
must-have
features
which
have
been
integrated
into
the
operator
to
fulfill.
Our
basic
needs
needs
that
I
just
presented.
So
first
is
self-healing
clusters.
Without
this
feature,
the
operator
doesn't
make
sense
and
the
operator
has
to
continuously
reconcile
and
solve
for
application
issues
also
high
highly
available
reads.
A
This
is
true,
and
more
more
nodes
are
available,
also
virtual,
highly
available
writes.
This
provides
us
minimum
of
downtime
due
to
fast
failover,
we'll
see
how
replication
lag,
detection
and
mitigation
the
operator
takes,
takes
lag,
node
out
of
rotation
or
unhealthy
nodes
and
resource
abuse
control.
This
is
very
useful
because
it
can
limit
noisy
queries
that
can
slow
down
your
cluster,
also
automate,
backups
and
restore
this
one
speaks
for
itself,
so
all
of
this
feature
have
proven
to
be
very
helpful
compared
with
our
setup.
A
Now,
let's
move
on
to
the
practical
aspects
of
building
the
operator
in
these
figures
presented
the
entire
system
overview,
the
heat
actually
split
into
three
main
parts:
the
control,
plane,
data,
plane
and
monitoring.
The
control
plane
consists
of
the
operator
and
its
components
and
are
deployed
using
count
using
usually
in
a
dedicated
namespace.
A
Here
we
have
the
controller
itself
and
your
Orchestrator
a
high
availability
and
application
management
tool.
I'll
come
back
to
it
later
in
the
talk
now,
the
data
plane
represents
the
Mexico
actual
Mexico
deployment
and
it's
made
of
basic
components:
resources
like
pods
services
and
so
on,
which
can
be
spread
across
multiple
namespaces
and
last
but
not
least,
we
have
monitoring,
which
is
performed
by
parameters
the
standard,
kubernetes
monitoring
system
going
deeper
into
the
data
plane.
We
can
see
that
the
my
cycle
cluster
has
multiple
components.
A
For
example
the
state
who
said
that's
the
main
component
and
is
responsible
for
provisioning,
pods
and
persistent
volumes
for
each
mexico
node.
Also,
there
are
two
services,
a
master
service
that
always
points
to
the
master
master
node
and
a
healthy
node
service
that
points
to
all
nodes
that
are
considered
healthy
by
the
operator.
A
Some
of
them
are
based
on
Percona
toolkit,
which
are
responsible
for
serving
several
action
like
lag
detection,
monitoring,
resource
limit
policy
enforcement,
and
there
is
more
one
more
container
that
provides
an
endpoint
for
backups
and
for
other
nodes
to
initialize
from
so
now.
I
want
to
mention
some
specific
challenges
that
we
had
during
the
implementation
of
the
operator,
such
of
Orchestrator
integration.
A
How
we
integrate
Orchestrator
a
third-party
tool,
so
we
don't
have
to
reinvent
the
wheel,
persistent
volume
cleanup.
We
have
to
manage
persistent
volumes
or
self,
because
the
Wake
Burnett
is
does.
It
is
not
fit
suitable
for
mass
equal
operator
upgrades
and
deploy,
which
is
a
common
problem
between
the
operators
because
and
provides
very
modest
CRD
management
and
my
sequel
operate.
A
A
Orchestrator
is
a
sub
component
of
the
entire
operator
and
it's
the
tool
that
handles
my
sequel,
fan,
overs
and
topology.
But
it's
not
meant
to
be
stateless,
as
operator
usual
usually
are
so
kubernetes
has
a
state.
Orchestrator
has
a
state,
and
the
operator
doesn't
know
which
one
to
listen
to
here.
I'm
talking
about
an
information
flow
conflict.
To
fix
this,
we
choose
to
implement
a
reconciliation
loop
between
Orchestrator
and
communities,
which
every
few
seconds
reconcile
reconciles
the
state
between
them.
A
So,
on
one
hand,
Orchestrator
is
responsible
for
updating
replication
topology
in
emergency
situation
in
case
of
downtime
and
to
observe
the
current
status
of
the
cluster.
On
the
other
hand,
the
operator
reconciles
the
desired
topology
into
the
orchestrator
and
provide
service
discovery
for
Orchestrator.
So
even
if
the
data
from
Orchestrator
is
lost,
the
operator
is
able
to
restore
that
data.
A
The
magical
data
is
stored
in
persistent
volumes
managed
by
the
state
Fassett,
but
this
implies
that
when
a
cluster
is
scaled
down,
the
volumes
are
not
deleted.
So
after
a
while
when
the
cluster
is
scaled
back
up,
the
replication
may
fail
to
fix
this.
We
choose
to
implement.
We
choose
to
implement
cleaner
ideal,
a
worker
that
deal
its
persistent
volumes
when
the
cluster
scaled
down,
except
for
node
0,
which
is
a
special
case,
because
that
data
should
be
kept
as
longer
as
longer
as
the
cluster
exists
to
preserve
data
to
preserve
losing
data.
A
Another
challenge
is
a
common
problem
in
the
world
of
operators,
because
how
share
D
management
is
still
very
painful.
Currently,
the
de
facto
standard
for
packaging
application
is
helm.
If
you
are
a
hand
user,
you
probably
know
that
CRD
management,
still
still
very
painful
and
ham,
doesn't
know
a
dozen
provides
and
obligated
path
for
series.
What
is
more,
my
sickle
operator
is
still
in
development
and
share.
This
specification
are
subject
to
change.
A
This
made
us
to
install
er
this
without
validation
to
minimize
user
intervention
at
upgrades.
However,
we
hope
we
hope
that
this
is
a
temporary
fix
solution
until
hands
improve
its
support
for
CR
DS.
Now,
a
specific
challenge
is
how
my
sequel
upgrades
are
performed.
Kubernetes
already
provides
us
some
upgrade
policies
like
rolling
up
gates
for
stateful
set,
but
this
is
not
exactly
gentle
for
my
sequel
because
it
can
choose
to
update
first,
the
master
which
will
trigger
a
fellow
to
the
replica.
A
A
We
try
also
other
techniques
like
pod
finalizer
to
block
the
pod
deletion
until
the
failover
is
done,
but
we
hit
a
dead
end
because
we
misunderstood
how
kubernetes
finalizer
work
it
happens.
Also,
we
try
using
container
lifecycle
hooks
but
proved
to
be
too
complicated
to
implement,
so
we
choose
to
implement
on
delete
policy
which
is
still
work-in-progress
now
I
want
to
share
with
you
the
current
status
of
the
project
and
some
future
plans.
A
A
We
have
a
good
feedback
from
community
so
far
and
some
major
platforms
already
actively
use
it
and
contribute
to
it.
So
I
would
like
to
invite
you
to
visit
the
project
page
on
github
and
for
any
questions
to
join
on
slack
Channel.
My
sequel
operator,
slack
channel
coming
up
next
I
will
play
a
video
or
recorded
video
with
the
second
part
of
the
presentation
where
session
talked
about
a
case
study
how
they
migrate
from
cloud
manage
sequel
to
govern
others
manage
so
that
our
session.
That's.
B
Hello,
my
name
is
Sachin
I
work
as
an
engineer
at
platform.
9,
sorry
I
couldn't
be
there
at
Q
Khan
in
person,
but
hopefully
this
video
would
give
you
you
enough
insight
into
how
we
have
been
using
this
operator
and
what
your
our
clients
are
with
this
operator
and
I'll
be
around
to
answer
questions
over
phone
as
well.
So
just
a
quick
intro
to
platform
9.
We
are
a
company
which
offers
open
cloud
services
store
customs
and
we
have
an
offering
which
is
based
on
OpenStack
as
well
as
kubernetes.
B
B
So
what
is?
How
does
this
try
to
the
my
Sequoia
operator?
We
started
back
in
2014
and
as
well
startup.
We
adopted
the
public
cloud
model
so
using
public
cloud
services
without
investing
too
much
into
our
IT
infrastructure,
be
equal
to
grow
our
customer
base
of
fairly
quickly,
but
at
the
current
scale
the
cost
of
public
cloud
is
quite
significant.
B
Just
our
database
bills
amount
to
tens
of
thousands
of
dollars
and
that's
a
significant
drag
for
a
small
sized
company
like
us.
So
as
I
explained
before
the
2014-2016
years
were
mainly
based
on
public
cloud
services
using
exclusively
RDS
or
all
our
diplomatic
work
needs.
We
also
used
other
services
like
the
database,
DNS,
load,
balancer,
etc.
There
are
two
Phillip
gaps,
then
in
2017
we
decided
to
move
completely
off
of
the
public
cloud
in
order
to
save
on
these
costs,
as
well
as
scale
according
to
our
requirements.
B
So
we
we
already
are
we
started
using
our
own,
offering
the
OpenStack
offering
to
run
our
the
instances
that
we
run
for
our
customers,
and
we
run
this
in
our
our
own
private
cloud
in
London.
But
at
that
time
we
realized
that
moving
off
of
database
wasn't
so
easy,
because
there
was
no
not
quite
equivalent
service
like
RDS,
that
you
could
simply
run
on
a
private
road
infrastructure
and
that's
when
we
found
this
my
sickle
operator
created
by
press
labs
and
tested
it
validated
it
and
we
are
rolling
it
out
in
production.
B
So
the
plan
is
that
this
year,
we'll
be
rolling
out
our
production
deployments
and
by
end
of
2020
we'll
have
all
our
production
deployments
on
this
our
database.
So
what
were
the
hurdles
to
to
running
our
own
database
as
a
service
and
getting
out
of
consuming
the
the
public
cloud
based
databases?
So
first
problem
was
that
the
our
deployment
automation
is
set
up
with
a
self-service
API
based
model
in
mind,
so
the
ansible
based
scripts
that
we
have
they
exercise.
B
The
RDS
api
is
to
create
scale,
delete,
resize
databases
and
we
needed
a
very
similar
self-service
api
experience
in
our
own
translation.
We
also
relied
on
snapshotting
and
backups
that
RDS
profiles
and
again
this
this
process
was
also
automated
using
AWS
api's,
and
we
needed
a
replacement
for
that.
So
because
of
these
two
points,
we
couldn't
really
adopt
a
solution
where
we
created,
we
create
a
database
in
a
VM
or
on
a
bare
metal
silver
and
then
ssh
into
it
to
make
changes
that
that
would
have
broken
our
automation.
B
Another
problem
was
that
by
relying
exclusively
on
a
hosted,
My
Secret
Service
me,
we
didn't
really
have
database
admins
to
save
us
from
situations
where
our
database
backup
goes
wrong
or
database
current
gets
corrupted
or
the
service
keeps
crashing
divison
bugs.
We
also
require
comparable
performance
with
audience,
because
we
we
wanted
to
sort
of
do
a
lift
and
shift,
and
we
wanted
to
make
sure
that
when
we
move
a
customer
to
an
on-prem
database,
they
shouldn't
see
any
performance
degradation.
B
B
So,
with
these
goals
in
mind,
we,
we
realized
that
using
a
simple
VM
based
approach
might
not
be
a
good
thing
for
us
and
we
started
looking
at
a
selfing
platforms
like
kubernetes
to
run
our
databases,
that's
what
we
found
out
about
the
my
sickle
operator
and
understood
various
features
that
it
provides
and
it
basically
ticked
all
our
check
boxes.
So
this
is
how
our
comparison
per
our
requirement
looks.
B
Like
looks
like
we
wanted
a
self-service,
simple,
open,
API
and
that
is
provided
by
the
CR
DS
that
could
that
this
operator
exposes,
and
it
hooks
nicely
into
our
automation.
We
wanted
a
drop-in
replacement
for
my
sequel
and
since
this
implementation
was
a
spark
owner
which
is
a
blue
one,
my
secure
database,
it
is,
we
found
it
to
be
quite
compatible.
We
haven't
really
discovered
any
breakages
or
bugs
in
our
services
to
do
switch
in
the
database
itself.
B
Since
we
are
trying
to
do
this
at
scale,
so
we
will
be,
will
have
around
300
databases
at
least
to
manage
when
you
go
to
production.
We
wanted
a
fairly
automated
and
loaded
solution
when
it
comes
to
handling
failures,
and
we
found
that
having
the
highly
available
database
using
Orchestrator,
it's
quite
appropriate
for
our
requirements
and
we
so
we
tested
out
straighter
in
the
wild
and
found
that
is
able
to
recover
from
TB
master
failures
in
a
quiet,
automated
and
seamless
manner.
B
And
last
but
not
least,
we
will
be
relied
on
the
built-in
monitoring
that
the
databases
service,
provided
we
regularly
check
and
generate
alerts
based
on
already
seconds
and
certain
thresholds
and
the
witness
matrix
that
my
sickle
operator
provides.
We
found
them
to
be
quite
adequate
for
this
purpose.
In
fact,
they
are
more
extensive
than
water
date
manage
database,
and
this
provides
so
this
database
checked
all
the
boxes
that
we
needed
and
we
started
implementing
it
on
thread,
and
this
is
how
our
infrastructure
looked
like.
B
So,
as
I
explained
before,
we
had
an
open,
stat
cloud
which
provided
multiple
availability
zones,
each
availability
zone
is
essentially
a
failure
domain
for
compute
storage
as
well
as
networking,
and
although
this
diagram
doesn't
show
it,
our
storage
is
also
multiple,
easy
aware
and,
for
example,
the
failures
in
easy
to
doesn't
impact
the
storage
in
easy
parking.
So
we
have
these
three,
as
is
provided
by
underlying
OpenStack
clock
and
for
kubernetes
week
we
use
the
multi
master,
our
deployment,
where
each
master
runs
in
its
own
availability
zone
and
we
have
a
load
balancer
in
front.
B
Two
rows
will
be
logging,
the
three
motors
to
deploy
databases.
We
also
leverage
this
AZ
concept,
so
the
availability
zone
labels
are
bubbled
up
as
work
on
all
the
labels
in
kubernetes,
and
the
deployment
of
any
database
always
has
at
least
three
instances,
so
one
master
and
two
replicas
and
each
one
of
them
come
up
in
a
different
abilities.
In
addition,
the
orchestrator
piece
also
runs
with
a
Z
multi
a-z
in
mind,
so
there
is
one
copy
of
Orchestrator
in
djz.
So
single
easy
failure
does
not
impart
all
the
layers,
including
the
kubernetes
Orchestrator.
B
As
well
as
the
database,
in
addition,
we
take
backups
of
this
database
to
s3
and
recover
from
a
stream.
So
that's
the
layer
of
cloud
infrastructure
which
still
using
so
the
current
state,
is
that
we
have
validated
and
it
works
pretty
well
for
audience
case
and
all
our
internal
Devon
test
environments
are
use
this
database
today
to
rule
out
the
operator
and
to
test
the
newer
version
of
operators.
We
have
set
up
three
clusters:
the
dev
cluster
is
used
to
validate
changes
in
operator
and
show
that
there
is
no
breaking
change
after
validation
and
testing.
B
So
our
plan
is
to
do
100%
deployments
of
databases
managed
by
a
bicycle
operator,
and
another
nice
thing
out
of
this
exercise
is
that
we
we
stumbled
across
the
operator
paradigm
and
the
CR
DS,
which
is
very
powerful
for
managing
your
infrastructure
in
a
fairly
automated
map.
So
we
plan
to
use
the
same
model
for
Oh
Matias,
based
monitoring,
drug
collection,
API
gateways,
and
so
with
that
I'm
end
of
at
the
end
of
my
presentation.
Thank
you
and
I'll
be
around
for
questions.
A
C
E
A
We
have
finally
did
I
rewrite
something
we
change
and
it's
a
release
candidate
and
we
have
to
improve
some,
as
I
said,
orchestrate
our
integration
and
to
finish
on
delete
policy
implementation
and
after
that,
almost
ready
from
our
point
of
view.
Of
course,
there's
still
need
for
a
lot
of
tests,
but
we
already
use
it
in
production
and
such
guys
from
platform
lines
already
test
and
used
the
operator
in
I
think
in
production,
not
sure
for
some
of
them
workloads.
C
D
All
right,
I'll
tell
you
one
that
oh
yeah,
so
here
for
almost
a
year
now
and
it's
been
pretty
solid
in
replacing
audiences
this
week,
:
tortillas.
That
gives
us
a
lot
of
data
that
are
this
provides.
We
can
monitor
it
pretty
well
most
of
the
problems
we
we
had.
We
have
not
actually
because
of
the
operator,
but
because
the
way
our
content
from
kubernetes
was-
and
so
we
think
it's
Russian
ready
and
as
I
said
in
the
talk
we
have,
we
hope
that
we'll
be
able
to
roll
it
out
completely.
F
Thank
you
very
much
for
the
talk.
I
have
a
question.
So
first
one
is:
how
do
you
deal
with
growth
when
you
have
to
change
the
precision
volume
claim,
or
does
that
work
with
the
operator
and
the
second
one
is,
since
you
delete
the
persistent
volume
when
you
scale
down
when
you
go
up
again,
you
have
to
replicate
all
the
data
and
what's
the
latency,
there.
A
A
Yeah
you
can
resize
the
persistent
volumes
the
operator
doesn't
do
that,
so
you
have
to
do
it
manually,
it's
Cooper
necessary.
It
provides
you
Matt
to
resize,
yeah,
okay,
I,
wonder
if
you
could
elaborate
on
why
they
will
be
interested
in
moving
to
proxy
executors
and
still,
instead
of
having
two
services
to
make
easier
to
to
make
to
make
it
easier
for
people
for
application
to
connect
to
it.
It's
easier
to
have
a
one
endpoint.
Instead
of
using
some
I,
don't
know
specific
publication
logic.
A
A
G
A
G
A
Orchestrator
is
highly
available.
It
can
be
deployed
in
multiple
clusters,
so
it
can
be
highly
available.
It's
used
raft,
communicate
and
so
on.
If
it's
highly
available,
then
it's
always
up.
Hopefully,
if
not
that's
for
the
moment,
it's
an
undefined
behavior
it.
We
still
do
tests
to
see
what's
happened.
B
C
C
D
Yes,
sure
so,
if
usually
when
we
resize
the
database,
we
we
do
have
some
interval
of
time
where
we
can
take
the
environment,
offline,
resize,
the
resources
or
the
database
and
bring
it
back
up,
and
it
works
pretty
well
with
the
operator
in
our
case
or
the
orchestrator
piece.
As
I
mentioned,
we
have
multiple
availability
zones
and
each
replicas
in
the
schedule
site
of
Orchestrator
is
spun
off
in
its
own
availability
zone.
So
we
are
really
banking
on
the
fact
that
failure,
one
availability
zone
does
not
impact
either
the
database
or
the
orchestrator
Orchestrator
workflow.
D
We
have
had
issues
in
the
past
where
our
networking
went
down
and,
as
a
result,
one
of
the
issues
was
unavailable
and
we
we
were
able
to
successfully
have
the
orchestrator
revert,
the
master
replica
master
to
a
replica
of
the
database,
and
this
this
depends
on
the
time
outs.
Certain
the
orchestration
and
the
setting
we
currently
have
is
about
it
waits
for
about
three
minutes
before
changing
the
master
of
replicas.
D
F
A
We
don't
do
that
yeah,
you
have
to
change
specification
and
that
will
trigger
a
rolling,
upgrade
and
update
update
if
you
change
the
requirements
limits
and
so
on.
On
pods,
you
have
accessed
them
Mike
on
my
sequel,
cluster
CRD
and
you
can
modify
them
and
that
will
trigger
the
deploy.
It's
not
online.