►
From YouTube: DBaaS on Kubernetes: Peanut Butter and Jelly, or Oil and Water - Steve Croce & Wes Ashkinazy
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
DBaaS on Kubernetes: Peanut Butter and Jelly, or Oil and Water - Steve Croce & Wes Ashkinazy, ObjectRocket
Kubernetes wasn't designed to run databases, but recent features have opened the door to stateful applications and databases in particular. However, once you add the "as a service" part, things get a whole lot more interesting. Steve Croce and Wes Ashkinazy from ObjectRocket talk about the problems they expected, the challenges that surprised them and ultimately the solutions that allowed the ObjectRocket team to design Elasticsearch, MongoDB, and Redis as a service on Kubernetes. The session will cover a set of particular challenges in the design of custom Kubernetes operators, solving problems with Event Recorder, Remote Execution, and Stateful Set Update Strategies, to name a few, and leveraging other CNCF projects to fill in the gaps.
To learn more: https://sched.co/Grap
A
A
Just
a
quick
intro
if
you're
not
familiar
with
object
rocket,
we
are
a
database
as
a
service
company.
We
currently
host
MongoDB
elasticsearch
with
cabana
and
Redis.
Our
real
focus
is
on
support,
so
we
attach
you
know,
24
by
7,
by
365
support
for
pretty
much
every
customer
there's.
You
know
no
upsell
there
and
really.
Our
goal
is
to
provide
free
migrations,
free
upgrades
all
that
stuff
to
basically
be
your
database,
ops
team.
So
just
a
quick
intro
on
who
we
are
ok.
A
That
was
probably
the
you
know,
the
best
game
in
town
for
us
and
everything
was
custom.
When
we
built
this,
we
had
a
goal
at
the
beginning.
In
the
very
beginning,
the
company
to
be
kind
of
the
absolute
fastest
hosted
MongoDB
MongoDB
was
the
lead
product,
and
so
everything
was
custom
built.
It
all
runs
on
our
own
bare
metal,
bare
metal
environment.
But
since
then,
you
know
six
years
is
a
long
time
in
technology.
A
lot
to
change.
There's
a
lot
more
available
to
us.
Docker
has
now
become
the
de-facto
container
formats.
A
There's
a
lot
of
opportunity
for
us
to
move
off
what
we
had
and
then
also
custom
orchestration.
Just
there's
no
reason
for
us
anymore
to
do
it.
Things
like
Cooper
net
kubernetes
have
gotten.
You
know
really
capable
enough
that
there's
nothing.
We
could
do
ourselves.
That
would,
you
know,
be
able
to
be
what
what
we
get
from
the
community
today
and
then
the
final
piece-
and
this
is
the
you
know-
one
of
the
real
drivers-
is
we
needed
to
be
able
to
stamp
out
our
environment
in
any
cloud.
A
We
wanted
to
the
bare
metal
environment,
we're
in
it's
great
for
the
customers
we
had
today
that
want
you
know
the
the
absolute
fastest
thing,
but
for
everybody
else
we
get
a
lot
of
requests
to.
You
know
once
again
move
our
databases
closer
to
the
application.
So
that's
the
quick
you
know
kind
of
outline
of
why
we're
moving
everything
and
kind
of
building
a
brand
new
platform,
and
so
this
is
kind
of
a
little
nod
to
the
rest
of
the.
A
You
know,
rest
of
the
presentation
and
we'll
go
through
this
learning
curve
we
had
where
and
I'll
talk
will
talk
about
when
you
know
when
kubernetes
was
easy,
when
kubernetes
wasn't
easy
and
how
it's
getting
easy
again
and
you'll
see
on
the
top
right,
we'll
have
some
of
these
little
gauges
and
so
the
green
they're,
the
areas
where
kubernetes
actually
solved
our
problem
really
easily.
There's
the
Reds,
where
you
know
we
had
to
kind
of
invent
something
to
solve
a
problem
but
yeah
you
will
go
through
this.
B
So
little
context
when
you're
looking
at
kubernetes
and
these
orchestration
platforms
for
hosting
stateless
services,
Web
Apps
workers,
things
like
that,
we
were
all
excited
when
we're
talking
about
putting
stateful
applications
on
there.
A
lot
of
red
flags
hid
from
the
dev
team
a
year
or
two
ago,
a
lot
of
these
platforms,
weren't,
really
designed
or
up
to
snuff,
yet
to
run
stateful
applications.
B
We
all
know
databases,
don't
really
tolerate
failure
that
easily
some
better
than
others
and
operators
really
we're
still
just
becoming
a
thing
to
work
with
and
just
for
some
context
about
what
our
platform
kind
of
looks
like
today.
We're
gonna
have
an
infrastructure,
API
we've
centralized
logging
monitoring
and
use
on
boy
frickin
grass.
We've
got
some
fluency,
some
weave
works.
We
have
clusters
in
multiple
regions
in
our
alpha
program
and
with
all
of
that,
we're
just
focusing
on
this
kind
of
small
bucket
today
and
how
we
manage
a
stateful
database.
B
In
our
platform,
the
presentation
is
gonna,
follow
a
flow,
so
we're
gonna
start
with.
All
of
these
combined
are
the
table
stakes
to
really
host
a
database
as
a
service.
We've
got
the
basics,
which
is
kind
of
how
we
create
and
manage
clusters,
there's
security
for
those
clusters,
database,
administration
and
automated
tasks
that
our
database
needs
and
then
really
the
service
features
that
take
it
above.
Just
managing
a
database
for
somebody.
A
We're
just
gonna
have
all
three
perform
all
the
roles.
We
have
some
endpoint
that
we
offer
that
load
balances
between
those
three
nodes,
but
then
off
to
the
side
we
also
host
cabana,
which
is
a
visualization
package.
It's
part
of
the
elastic
stack.
We've
got
some
other
dashboards
that
are
available
as
site
plugins,
which
also
need
external
access.
But
then
we
also
have
some
other
tools
that
we
run
internally
so
curators
in
an
index
management
tool
and
a
last
alert
as
an
open
source
alerting
tool.
So
that's
kind
of
the
base
cluster.
A
When
you
create
a
cluster
and
when
we,
you
know
kind
of
we
host
something,
this
is
kind
of
the
default
stamp.
But
then
you
can
add
on
to
that.
You
can
add
more
nodes.
You
can
add
more
specific
nodes
and
specific
roles.
You
can
also
add
other
tools
like
log
stash,
which
are
things
that
will
all
support
once
we,
you
know
kind
of
once.
We're
fully
live
with
the
platform.
A
So
the
thing
that
makes
all
of
this
you
know
kind
of
possible
is
operators,
and
you
know
anybody
if
you've
all
been
here.
The
last
couple
of
days
and
watch
the
talks
this
morning.
There's
plenty
of
overviews
about
operators
and
so
I'm
not
going
to
go
into
too
much
details
a
detail
about
an
operator,
but
just
from
a
high-level
overview.
You
know
not
a
deep
dive.
Anybody
who's,
not
familiar
operator
gives
us
the
ability
to
basically
create
custom
objects
within
within
kubernetes.
A
If
you
look
at
the
left
side,
you
know
this
is
just
an
example
of
what
our
custom
resource
looks
like
it's
got
a
you
know
to
search
multi-role
kind,
which
kubernetes
doesn't
know
what
an
elasticsearch
multi-role
is.
So
we
have
to
tell
it.
We
create
a
custom
resource
definition
that
kind
of
defines
what
fields
and
what,
how
that,
how
that
a
manifest
is
going
to
look,
but
then
also
our
controller
can
then
take
that
manifest
and
turn
it
into
things.
A
Once
again,
the
kubernetes
understands
so
turns
it
into
some
stateful
sets
some
services,
some
secrets,
some
cron
jobs
and
lots
of
other
things
and
then
also
perform
some
custom
tasks
along
the
way.
So
once
again,
not
a
huge,
you
know
deep
dive
on
operators,
but
hopefully
enough
of
a
groundwork
that,
if
you're
not
familiar,
you
know
you
can
kind
of
keep
up
through
the
rest
of
the
conversation.
A
So
when
we
set
out
to
write
this
operator,
there
was
kind
of
three
you
know
kind
of
three
paths
we
had
or
the
three
paths
that
kind
of
made
the
most
sense
to
us.
We
could
use
an
off-the-shelf
kind
of
open
source
operator.
There
are
a
couple
of
good
elasticsearch
options
out
there.
We
could
use
a
library,
a
utility
library
there's
one
out
there
called
Cooper
that
looked
interesting
to
us.
That
would
allow
us
to
standardize
across
all
the
different
products,
because
once
again
we
also
have
you
know
a
MongoDB
team.
A
We
have
a
Redis
team,
we
have
other
teams,
so
that
was
an
option.
The
third
option
was
to
just
build
from
scratch:
just
go
and
build
our
own
operator
now,
given
that
we,
you
know,
we
were
a
lot
of
the
people
at
the
company
were
new
to
kubernetes.
When
we
kicked
this
project
off
earlier
this
year
we
went
with
off-the-shelf
operator
for
the
POC,
so
we
took
the
most
capable
operator
that
we
could
find
in
the
community
and
used
that
for
our
POC
and
learned
a
lot
and
what
we
really
found
was.
A
It
did
some
of
the
things
we
want.
It
needed
a
little
bit
more
functionality,
but
the
real
problem
and
the
reason
that
we
couldn't
use
it
to
go
into
production
or
we
decided
to
kind
of
make
a
break
going
into
production.
Was
it
was
developed,
a
couple
of
versions
of
kubernetes
back
and
so
kind
of
refactoring
the
whole
operator
to
get
it.
You
know
get
it
up
to
speed
and
using
the
latest,
things
ended
up
being
a
pretty
big
burden
and
we
would
have
to
fundamentally
change
the
way
of
that
operator
work.
A
So
it
kind
of
made
more
sense
to
us
at
the
time
to
really
just
start
from
start
from
scratch
and
for
production
build
our
own
build.
Our
own
operator
gave
us
a
little
bit.
More
control
gave
us
a
little
bit
more
freedom,
and
it
just
seemed
like
it
was
a
better
opportunity
than
to
have
to
go
once
again
take
somebody
else's
operator
and
completely
fundamentally
change
it.
A
Then
somebody
releases
something:
that's
not
you
know
solves
it
better
and
you
have
to
kind
of
make
the
call
as
to
whether
you
go
back
and
every
factor
or
you
just
move
on
and
then
the
final
piece.
Another
piece,
that's
just
gonna
be
super
high
level,
since
everybody
else
has
done
deep,
dives
on
it.
For
you
know
a
long
time
now,
you
know
we
couldn't
do
what
we
do
without
stateful
sets
it's
you
know
we
need.
A
You
know
persistent
storage
with
all
of
our
with
all
of
our
pods,
so
obviously
stateful
sets
provide
that,
but
also
stateful
set
to
provide
a
whole
bunch
of
utilities
about
how
you
deploy,
how
you
scale
and
how
you
restart
your
resources.
So
once
again,
I'm
not
gonna,
you
know
that's
what
we'll
leave
it
that
and
spend
a
little
bit
more
time
digging
into
the
things
that
are
not
not
haven't
been
covered
as
well,
but
you
know
going
back
to
the
easy
meter.
A
You
know
kind
of
calling
that
out
making
the
operator
decision
was
a
little
bit
of
a
harder
problem.
Stateful
sets,
you
know,
kind
of
landed
in
the
middle
because,
as
you'll
see,
there
are
some
things
we
had
to
go
kind
of
do
our
own
work
to
make
stateful
sets
do
exactly
what
we
needed
for
a
database,
but
it's
getting
a
little
bit
easier,
ziz
pretty
much
the
story
here.
So
I'll
turn
it
over
to
Wes
for
security.
B
Right
so
now
we
have
a
basic
cluster
out
there.
It's
built
it's
running,
but
we
need
to
find
ways
to
run
authentication.
We
needed
to
secure
that
traffic
with
certificates
and
a
couple
other
things,
and
we
also
needed
to
worry
about
isolation
since
we're
hosting
it
for
a
company.
Here's
some
grumpy
cat,
so
I'm
gonna
go
with
namespaces
first,
so
namespaces
a
hole
in
kubernetes
are
kind
of
really
easy,
intro
concept.
B
Our
ingress
layer
is
how
customers
access.
It's
also
how
we
run
our
admin
tools
against
then
once
inside
that
nothing
is
gonna
get
in
or
out.
No
other
customer
is
gonna
impact,
someone
else
and
it
gave
us
a
really
easy
way
to
just
check
a
box
and
now
everyone's
secure
from
each
other.
On
top
of
that,
namespaces
are
also
gonna,
give
us
a
really
strong
isolation
with
secrets
and
config
maps.
Those
are
isolated
to
a
namespace.
B
So
when
we
create
unique
certificate
for
a
cluster
to
encrypt
communication,
isolated
from
that
namespace,
it's
hard
for
us
to
accidentally
dump
that
certificate
somewhere
else
or
give
someone
access
to
it,
leading
into
that,
and
why
part
of
these
certificates
are
important
is
how
we
manage
authentication
to
the
cluster.
Now
so
we've
isolated
it.
Now
we
have
to
make
sure
people
can
securely
connect
to
it.
Elasticsearch
on
its
own
does
not
give
you
an
authentication
mechanism
without
a
license.
B
There
are
a
couple
open
source
projects,
remote
search
guard,
they're
great
to
work
with
if
you're
looking
at
things,
they're
kind
of
cool
they've
got
a
good
product,
but
what's
important
for
this
part
of
this
toy
back,
is
they
have
an
admin
tool
that
you
need
to
run
in
order
to
apply
configuration
changes?
There's
no
API,
there's
nothing
like
that.
It's
run
a
command
applies
and
manifest
to
your
cluster
and
use
the
certificates
to
authenticate
for
us
to
do
that
at
scale
we
needed
to.
We
looked
at
a
couple
different
options.
B
There
is
a
centralized
service.
Oh
it's
a
very
common
practice.
You
distribute
workload,
you
connect
to
a
remote
machine.
You
run
off
the
queue
pretty
standard.
Then
we
looked
at
what
kubernetes
has
and
there's
a
job
workflow.
It's
really
great.
We
can
dictate
where
this
container
turns
on.
It
gets
access
to
these
isolated
resources,
but
you
have
to
wait
for
resources
to
spin
up
and
you
have
to
clean
up
your
old
jobs.
It's
not
really
a
repeatable
run.
Then
the
next
part
was
remote
command.
It's
essentially
just
us
a
teaching.
B
It's
like
running
a
remote
exec
command
with
SSH
in
your
container
of
VM
and
moving
along.
We
ended
up
going
with
remote
commands
because
it
allows
it
to
not
have
to
wait
for
resources
to
spin
up.
We
direct
we
directly
connect
to
the
container
where
the
tool
is
to
run
the
import
of
the
new
user
lists
and
update
it's
secure,
we're
just
connecting
over
the
Krupa
AP.
Is
it's
simple?
It's
lightweight
and
it
really
helps
solve
this
problem.
B
The
only
cons
were,
is
you
know,
in
my
opinion,
you
really
never
want
to
run
more
than
one
process
in
a
container
for
an
extended
period
of
time
here,
we're
kind
of
breaking
that
rule,
which
is
okay.
It's
a
30
second
run,
and
it
buys
us
so
much
more
and
it
was
kind
of
hard
to
kind
of
stack
overflow
our
way
around
how
to
get
examples
for
remote
come
in.
Not
a
lot
of
people
are
really
using
it.
B
A
Right
so
the
first
couple
pieces
are
once
again:
it's
you
know:
they're
the
bare
minimum
when
you're
spinning
up
a
new
cluster,
but
that's
you
know
a
lot
more
applicable
to
you,
know
kind
of
any
database,
not
necessarily
a
database
as
a
service
and
so
automating
the
administrative
tasks.
That's
really
where
you
know
kind
of
the
database-as-a-service
kind
of
pieces
start
to
click
in
as
well
as
the
service
features
that
Wes
will
talk
about
in
a
little
bit.
A
So
some
of
the
things
we
need
to
think
about
are
you
know
recurring
tasks,
things
like
backups
things
that
are.
You
know,
custom
configurations
things.
You
know
anything
that
usual
you
know,
database
administrator
would
do.
We
have
once
again
we're
very
support
heavy,
but
we
need
to
automate
this
when
you're
talking
about
hundreds
or
thousands
of
clusters,
you
can't
be
having
people
performing
manual
tasks
all
the
time.
So
that's
really.
A
The
goal
of
all
of
these
pieces
is
to
find
the
things
that
are
the
right
things
for
us
to
go
automate
make
them
part
of
the
platform
make
our.
So
you
know
make
our
support
folks
lives
better,
but
also
make
a
product
that
just
works
when
once
again
when
we're
offering
it
as
a
service.
So
the
first
thing
that
you
know
really,
you
know
made
this
easy
for
us
and
is
one
of
those
things
you
know
once
again
on
the
easy
meter.
We're
really
getting
into
the
green
zone
now
is.
A
We
can
figure
all
of
our
clusters
in
the
image
using
just
the
standard
settings
and
those
settings
are
generally
good
enough,
for
you
know
90
plus
percent
of
our
customers,
but
there
are,
you
know
there
is
a
subset
that
need
either
a
specific
setting
or
need
a
custom
plugin
or
need
some
other
things
well,
config
Maps
and
their
ability
to
be
mounted
as
a
volume
in
the
container
solve
this
problem
like
super
easy
super
easily.
So
first
example
is
elasticsearch
is
configured
via.
A
You
know,
llamo
file
that
goes
into
config
directory
for
most
of
the
common
there.
You
know
common
values,
we're
able
to
either
use
environment
variables
or
other
things
to
tune
some
of
those
settings,
but
for
things
that
are
brand
new
things
that
we
hadn't
baked
into
the
image,
we
can
actually
just
overwrite
that
entire
file
using
a
config
map
volume.
So
what
we
do
in
our
case
is
we
mount
that
file
in
its
own
directory
and
then
use
the
docker
entry
point
to
kind
of
move
that
file
into
the
right
space?
A
Now
you
know
anybody
who's
done
this
before
knows
you
can
go
overwrite
it.
You
know
a
single
file
in
a
directory
with
it
with
a
config
map
volume,
but
there
are
some
downsides
to
that,
and
so
we
wanted
the
flexibility
of
being
able
to
control
when
and
how
those
files
get
moved
around
and
it
kind
of
we
wanted
to
use
it
as
a
little
bit
more
of
a
general-purpose
way
to
put
files
in
place.
So
we
decided
to
use
a
script
to
move
things
in
to
the
right
place.
A
The
other
option
that
we
have-
and
this
is
for
things
where
we
wanted
to
just
drop
an
entire
directory
into
the
container-
is
the
second
option
here
is
to
just
take
a
config
map.
Take
the
files
in
the
config
map
mount
a
create,
a
volume
that
has
all
those
files
and
then
just
mount
that
volume
as
a
new
directory
somewhere
in
the
container.
There's
a
couple
of
examples
once
again
specific
to
elasticsearch
where
this
helps.
A
But
the
overall
idea
is,
you
can
take
any
file
you
want,
you
can
drop
it
into
a
directory
in
the
container
and
then
basically,
that
file
will
update.
You
know
kind
of
automatically
as
you
update
the
config
map,
so
it
works
out.
It
works
out
pretty
well,
we
don't
have
to
go
roll
containers
for
every
single
update
to
some
of
these.
Some
of
these
files
extremely
useful
once
again
super
easy.
A
It
was
something
that
was
a
little
bit
harder
on
our
older
platform
that
kubernetes
just
does
really
well
the
other
piece,
that's
key,
you
know
and
core
to
database
administration
is
the
recurring
tasks
and
so
two
examples.
We
had
we're
backups
right
every
day.
At
a
certain
time
or
some
range
of
times,
you're
gonna
take
it
back
up
and
you're
gonna
retain
some
number
of
backups
rather
than
having
to
you
know,
pick
a
job
scheduler
or
pick
some
kind
of
task
management
system.
A
Kubernetes
has
really
great
tools
built
in
with
respect
to
the
the
cron
job
system,
so
it
works
like
a
cron
job
works.
You
pick
a
time,
you
pick
an
action
and
it
does
that.
You
know
it
does
what
you
what
you
tell
it
to
do,
but
it
also
includes
some
nice.
You
know
kind
of
some
nice
settings
around.
Well,
what
have,
if
the
job
fails,
what
happens
if
the
container
fails?
A
That's
running
that
job,
and
so
in
the
first
example
on
in
the
case
of
a
backup
when
we
create
the
cluster,
we
submit
our
CR
and
what
our
controller
does.
One
of
the
things
that
is
an
output
of
that
initial
create
call
is
a
cron
job
that
runs
at
a
certain
time
and
takes
a
regular
backup.
We
just
use
a
curator
which
is
a
tool.
I've
talked
about
a
couple
of
times
to
do
our
snapshots
on
the
new
platform,
but
that
one
was
pretty
straightforward
and
pretty
easy.
A
However,
we
wanted
a
more
programmable
way
of
kind
of
creating
arbitrary
curator
tasks,
because
snapshots
are
pretty
standard.
You
take
them
at
a
regular
time.
Every
day,
it's
not
going
to
deviate
much
from
custard
customer
to
customer,
but
the
way
curator
works,
it's
very
specific
to
how
you're
using
your
data,
you
might
have
your
own
retention
policy.
You
might
have
aliases
that
you
want
to
manage
there's
a
lot
of
different
things.
A
Curator
can
do,
and
so
this
was
an
area
where
we
actually
created
a
new
customer
resource
definition
and
basically,
by
specified
by
submitting
a
custom
resource,
we're
able
to
create
arbitrary
curator
jobs
and
basically
using
the
cron
job
system.
What
we
do
is
provide
the
configuration
in
the
spec
of
the
CR
and
what
that
turns
into
is
a
cron
job,
with
a
config
map
mounted
as
a
volume
that
has
that
configuration
file,
that's
specified
in
the
original
CR.
A
So
once
again,
it
gave
us
a
nice
way
to
specify
these
arbitrary
curator
jobs
and
just
specify
the
action
file
and
the
configuration
for
that
for
that
run
of
right
in
the
matinee
know
the
initial
manifester
and
the
CR.
So
at
this
point
we
really
hit
our
stride.
You
know
it's,
it's
kubernetes,
it's
just
super!
You
know
it's
delivering
really
well
and
making
things
really
easy
for
us,
so
I'll
kick
it
over
to
West
for
service
features.
Now
sorry,
oh
that's.
B
Fine,
so
service
features.
So
now
we
have
a
very
basic
cluster
deployed
for
people.
They
can
access
it.
They
can
secure
it
we're
giving
them
backups,
there's
some
resiliency,
but,
as
everyone
knows,
when
you're
running
a
database
at
some
point,
you
need
to
scale
it.
You
need
to
head
storage,
you
need
to
add
more
members.
You
want
to
do
more
replica
sets.
B
You
need
to
update
it
to
get
over
security
fixes,
and
this
is
really
where
it
gets
to
the
next
level
and
we
end
up
having
a
whole
lot
of
conversations
with
product
of
he
actually
picked.
The
slide
I'm
just
kind
of
fantastic,
very
self-aware,
but
we
need
to
update
things
and
every
database
has
their
own
update
mechanism.
B
How
you
update
each
member
so
we're
gonna
circle
a
little
bit
a
back
around
to
staple
sets,
as
Steven
mentioned
earlier,
how
it
gives
you
a
rolling
update
mechanism,
but
before
just
gonna
bring
people
up
to
speed
a
little
bit
of
how
elasticsearch
updates
need
to
work.
Elastic
search
has
a
datastore,
you
shoot
a
note
in
the
head
or
turn
it
off
or
restart
it.
It's
gonna
try
to
self-heal.
It's
gonna
take
things
that
we're
not
primaries.
It's
gonna
turn
them
into
primaries.
It's
gonna
move
data
around
and
it's
usually
not
too
impactful
on
performance.
B
B
Do
that
within
five
minutes,
your
cluster
dies,
it
just
crashes,
we
tested
it
a
bunch
and
it
was
kind
of
fun.
So
stateful
sets
actually
do
give
you
an
ability
to
control
that
flow,
though,
and
it's
called
partition,
rolling
updates,
and
this
is
the
thing
where
it
took
a
little
bit
of
digging.
It
wasn't
really
documented
at
the
time
it
was
kind
of
a
newer
feature,
but
it
allows
you
to
just
set
a
value
of
an
ordinal
and
it
works
backwards.
B
So
you
can
say
my
stateful
set
to
the
five
I'm
gonna
start
at
five,
and
only
that
ones
getting
updated.
Then
you're
gonna
move
to
four
and
it's
gonna
get
updated
so
now,
you're
at
least
controlling
the
flow
of
when
a
resource
is
impacted
and
it's
a
little
less
detrimental
than
kubernetes
just
running
down
the
list
immediately.
B
It's
give
you
a
little
bit
of
context
of
what
a
more
complex
flow
is
than
just
restarting
a
cluster.
Sometimes
you
want
to
update
the
cluster,
so
if
you're,
updating
and
elastic
has
gotten
better
at
this
and
later
versions,
where
minor
and
patch
levels
it's
pretty
safe,
to
do
a
live,
update,
we're
actually
able
to.
We
need
to
be
able
to
update
that
member
verify
that
things
are
in
a
good
state,
maybe
run
some
analysis
tools
against
indexes
as
we
move
through
each
member
of
the
cluster.
B
There
are
times
that
we're
gonna
want
to
remove
a
node
from
the
cluster
when
you
change
the
size
of
a
stateful
set
kubernetes
just
sees
it
as
an
update.
It
would
just
delete
the
member
of
the
node
and
then
all
that
data
is
going
to
shuffle
around
elasticsearch.
We
have
to
say,
evacuate
this
node
and
monitor
that,
and
then,
when
the
note
is
empty,
we
can
actually
reduce
the
size
of
the
state.
We'll
set
a
couple
other
things
that
I
was
going
to
run
through
it.
B
Just
don't
remember
them
right
now
so
over
here
on
the
right,
you
just
see
kind
of
a
flow
of
what
I'm
talking
about
where
we're
disabling
allocation,
letting
the
update
run.
It's
gonna
update
one
member
of
the
pod
of
the
set
and
then
we'll
enable
allocation
verify
health
increase.
The
size
kubernetes
is
only
applying
it
to
the
newest
member
and
we're
good,
but
we
needed
to
manage
that
steam.
B
So
when
you're
interrupting
the
flow
and
running
these
tasks,
the
kubernetes
has
an
event
bus
that
you
can
query
on
really
busy
clusters,
though
that
becomes
an
expensive
query
you
can.
The
operator
can
fail.
We
write
all
of
these
updates
back
to
our
control
plane
that
we
can
query
out,
see
what's
happening
and
where
we
are,
but
that
becomes
expensive
that
every
time
the
operator
turns
on
it's
gonna,
clear,
every
single
cluster,
that's
in
that's
deployed
hit
our
main
API
and
then
see
what
the
state
it's
in
and
things
like
that.
B
We
ended
up
going
with
something
kind
of
simple
really
and
we
just
set
a
label
on
our
stateful
set,
with
the
actual,
with
the
event
that
is
currently
running.
So
when
we're
running
an
update,
we're
gonna
do
a
rolling
update
label
for
the
state
and
then
we're
gonna
set
the
ordinal
we're
gonna,
let
it
run
through,
and
then,
when
we
do
allocation,
we
set
the
label
to
allocation,
and
then
we
perform
another
action
we
set
with
that
action
is
so
if,
at
any
point
this
process
fails,
the
controller
is
going
to
come
online.
B
We're
gonna,
see
we're
in
an
update,
set,
we're
gonna
see
what
ordinal
we've
hit,
but
then
our
controller
is
also
going
to
know
what
we've
done
to
the
cluster,
and
then
we
do
that
by
building
a
giant
state
map,
the
state
map
is
kind
of
an
is
just
it's
annoying,
it's
big,
but
it
was
the
best
way
we
could
solve
the
problem
and
it
was
challenging,
as
you
can
see
this
a
little
bit
more
on
the
run
and
a
state
map
is
gonna.
Look
like
this.
B
Every
time
we're
gonna
run
an
update
on
a
member
of
a
stateful
set,
we're
gonna.
Look
at
what
that
label
is
we're
gonna
run
that
label
against
an
entire
set
of
switch
statements
based
on
what
that
is,
we're,
gonna
call
a
function,
perform
an
action
and
then
update
the
label.
I'm
just
gonna
keep
iterating
down
the
list.
This
specific
flow
is
just
the
first
part
which
says
I'm
in
an
update.
B
A
B
Wide
down
to
there
and
we're
gonna
end
up
having
to
revisit
how
to
do
this,
but
it's
been
working
for
now.
I
just
think
we're
gonna
hit
a
problem
so
might
not
be
the
takeaway
here
is
don't
do
this,
maybe
we're
actually
looking
at
sub
resources
which
have
come
out
which
are
giving
us
a
little
bit
more
granularity
on
specific
resources
of
states
of
stateful
sets
and
things
like
that
yeah.
This
was
kind
of
challenging
it's
kind
of
hard
to
keep
in
our
head
and
with
that
we're
gonna
move
on
to
things.
B
A
And
so
now
that
we
have
the
labels
on
it,
you
know
we
had
the
idea.
You
know
the
all.
The
things
guys
come
out
of
the
woodwork
and
it's
gonna
be
super
easy
and
it's
just
gonna
work
and
you
know
kubernetes
does
everything,
and
obviously,
once
we
dug
in
you
know,
we
didn't
know
we
didn't
know,
and
so
you
got
that
first
dip
right.
A
But
then
you
know
once
you
start
to
learn
all
the
facilities
and
all
the
tools
that
are
baked
into
you
know
baked
into
kubernetes
and
you're
keeping
up
with
all
the
new
releases.
There's
a
lot
of
options,
and
so
it
got
easy
again
right.
We
hit
the
basics,
it
got
really
easy,
and
so
we
hit
our
stride.
But
then
you
know
to
west's
point
with
the
state
management.
A
You
kind
of
hit
that
you
know
you
start
to
hit
the
things
that
maybe
there
just
isn't
a
tool
available
to
solve
it,
for
you
or
maybe
you
just
have
to
go,
write
your
own
code
or
your
own
workaround.
To
do
it.
That's
when
things
got
difficult
again,
but
now
what
we're
starting
to
see
is
now
that
we've
been
through,
that
we've
learned
a
lot.
You
know
we're
kind
of
hitting
the
upswing,
but
one
of
the
things
we'll
talk
about
in
a
second
is
also
it's.
A
Like
you
know,
you
blink
your
eye,
there's
a
new
feature.
That's
out
now,
out
with
how
quick
the
release,
like
you
know,
really
cadance
is
on
kubernetes
we're
finding
it
actually
makes
sense
to
you
know
the
longer
you
stay
in
the
game,
the
easier
things
are
gonna
get
just
by
the
strength
of
the
tools.
A
The
other
piece
that's
been
kind
of
interesting
to
us
throughout
this
is
you
know
we
tried
not
to
repeat
ourselves,
so
we
tried
to
be
super
dry
when
we,
when
we
created
the
you
know
the
first
CRD
and
our
first,
you
know
and
we're
submitting
our
first
CRS
to
create
our
resources.
The
idea
was,
let's
just
create
this
massive
CR
that
does
all
the
things
and
then
basically,
depending
on
all
the
different
things
we
want
to
do
in
our
cluster.
We
just
you
know,
provide
different.
A
You
know
different
versions
or
not
different
versions,
different
different
settings
in
the
in
the
CR.
What
we
found,
though,
is
it
made
the
code
really
dry,
but
it
was
super
complex
and
we
had
to
handle
a
bunch
of
different
conditions,
a
bunch
of
different
states
that
made
just
the
code
harder
to
read
harder
to
manage.
It
made
life
a
little
bit
harder.
A
However,
then
you
know
about
halfway
through
this
process
we
started
looking
into
just
you
know,
just
create
a
new
CRD
or
just
create
a
new
CR,
for
you
know
for
all
the
different
things
now
the
downside
of
that
is.
We
do
have
to
duplicate
some
code.
There
are
some
things
that
you
know
we
could
be
more
dry
about,
but
the
code
has
become
a
lot
a
lot
simpler.
It's
a
lot
easier
to
learn
it's
a
lot
of
usually
to
ramp
on,
and
it's
actually
a
lot
easier
to.
You
know
kind
of
debug
and
work
through.
A
B
Know
I'm
like
a
Steven
mentioned,
so
we
Eddie
curator
to
our
alpha
just
recently,
and
that
was
not
on
my
quarter
plan
originally.
So
he
just
came
is
like
we
have
a
health,
a
customer.
We
need
to
manage
their
instance,
their
indexes.
Let's
get
curator
out
there
simple,
so
it
was
an
afternoon's
to
two
days
of
work
to
just
spin
up
another
quick
CR.
We
already
have
a
bunch
of
CRS
that
do
this
and
just
make
it
a
little
more
specific
tune.
B
It
a
bit
more
to
just
manage
a
cron
job
and
it
take
a
config
map
and
it's
lettuce
really
iterate
very
quickly
on
new
features
and
respond
to
help
or
criticisms
and
actually
requests.
But,
like
Steve
said
it's
gonna
bite
us
any
ask
quite
a
bit
because
now
we
have
I,
think
six
see
ours
and
four
of
them
are
really
similar.
B
So
we're
looking
at
some
refactoring
as
some
holiday
homework
across
the
team,
and
then
we
tie
in
to
the
velocity
of
kubernetes-
and
it's
touched
a
few
points
where
massive
state
machine
great
idea,
nine
twelve
months
ago-
probably
not
a
great
idea.
Now.
We've
had
that
complexity.
Another
great
example
so,
from
the
beginning
and
I
alluded
to
it
before
and
how
we
update
resources.
We
need
a
way
to
expand
the
storage
on
these
EBS
volumes
that
were
attaching
to
these
pods
a
year
ago.
There
was
no
automatic
magic
way
to
do
that.
A
So
you,
you
know
you
all
can
try
it
there.
We
have
an
evaluation
program
right
now,
we're
kind
of
right
between
the
alpha
and
beta.
It's
not
yet
an
open
beta,
but
you
know
by
contacting
us,
we
can
give
you
some
free
resources
to
check
out
the
the
new
service.
You
know
kick
the
tires,
provide
us
with
feedback.
Give
us
some.
You
know,
give
us
some
ideas
on
how
you
think
we
can
make
the
product
better.
A
B
So
we
started
the
project
very
much.
We
want
to
be
open-source
we're
using
all
these
open-source
tools
and
then
we
had
to
start
making
decisions.
Oh
well.
We
need
this
component
to
send
messages
to
this
part
of
our
control
plan.
They're
like
well,
it's
a
hundred
lines
of
code
here
or
it's
a
brand-new
service
here.
So
we
started
just
getting
lazy.
B
What
we
do
we
are
committing
to
in
q1.
We
are
going
to
try
to
get
more
of
our
more
of
the
tooling
that
we're
using
out
there
and
visible
to
the
community.
We've
had
pain
points,
we've
learned
a
lot
and
we've
learned
a
lot
from
other
people's
projects.
So
we
want
to
get
that
out
there
and
one
of
the
reoccurring
themes
in
the
next
quarter
is
decoupling.
B
Some
of
our
business
logic,
decoupling
some
of
our
control,
plain
logic
and
being
able
to
open
source
this
sometime
next
year
early
next
year,
so
we
can
get
feedback
and
make
it
a
better
product,
everybody
and
then
we're
hiring
engineering
sales
full-stack
des
GUI
devs.
If
you
know
kubernetes,
go
any
of
those
things.
If
you
think
this
is
interesting
or
we're
mildly,
interesting,
come
and
talk
to
us,
we've
got
a
couple
more
guys
from
York
down
here
in
the
front
row.
I'm
sure
you
could
ask
them
some
questions
too
yeah.
B
B
Yeah,
so
we're
not
using
any
local
storage
and
we're
not
on
Prem
right
now
we're
doing
EBS
volumes
in
Amazon
and
we're
looking
at
a
shirred.
You
know
remote
disks
in
Azure
when
we
get
there
and
Google
we're
still
researching,
though
after
seeing
what
Hoover
talked
about
today
with
their
storage,
we're
gonna
start
investigating
that
for
our
really
high
performance
customers,
it's
just
harder
to
move
resources
around
when
you
lose
a
node
and,
like
you
saw
like
I
said
elastic,
does
not
handle
that
very
well
on
a
big
cluster,
especially
busy
ones.