►
From YouTube: Vitess at HubSpot: How We Moved Hundreds of MySQL Databases into... - Tom Petr & Alex Charis
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Vitess at HubSpot: How We Moved Hundreds of MySQL Databases into Kubernetes - Tom Petr & Alex Charis, HubSpot
In early 2017, HubSpot faced a growing problem: our MySQL infrastructure was complex, expensive, and failing to scale. Hundreds of databases running on more than a thousand EC2 instances were managed by just one person, and developers were getting fed up and began to migrate to other higher performance but non-relational datastores. We surveyed the landscape and decided to address these challenges by adopting Vitess and Kubernetes. This talk will detail our journey towards MySQL cloud-native nirvana. We'll cover lessons learned, bugs squashed, and tools built -- including the design and operation of our very own Kubernetes operator for Vitess. Attendees will walk away with a better understanding of what it takes to reliably manage and scale MySQL databases in Kubernetes, and how Vitess is a great tool for the job.
To learn more: https://sched.co/Gras
B
A
And
this
is
the
story
of
how
we
moved
hundreds
of
my
sequel
databases
into
kubernetes.
So
what
is
how
spot
we're
marketing
sales
and
services
software
platform
based
in
Cambridge
Massachusetts,
the
HubSpot
app,
is
actually
thousands
of
different
micro
services
built
and
deployed
hundreds
of
times
a
day
by
many
small
autonomous
teams.
Our
computing
footprint
is
about
3,000
v,
vm
s
in
both
AWS
and
g
CP,
and
before
you
get
to
the
good
stuff.
I
want
to
turn
the
clock
back
to
2013
for
a
minute.
Well
is
a
developer
at
HubSpot,
it's
kind
of
hard.
A
We
had
a
clunky
CI
pipeline
as
basically
just
one
Jenkins's
instance
that
we
affectionately
referred
to
as
big
J.
It
was
constantly
falling
over.
We
had
a
flaky
ssh
based
deployment
system
with
no
sophistication.
We
have
no
elasticity
and
no
automation
in
our
infrastructure,
which
meant
it
was
not
uncommon
to
be
woken
up
at
3:00
a.m.
in
the
morning,
because
someone's
nightly
job
was
taking
down
your
web
service
or
one
of
the
ec2
instances
that
powered
your
web
app
went
impaired
and
you
had
to
do
something
to
replace
it.
A
We've
been
looking
at
Apache
mezzos
at
the
time
and
we
seriously
considered
evaluating
it
in
the
fall
of
2013.
The
thing
about
mesas
is
different
from
kubernetes
is
that
it
only
provides
the
primitives
for
running
your
apps
on
a
cluster
of
resources.
Don't
worry,
I'm
gonna
get
to
careen
Eddy
soon.
This
is
I,
didn't
think
I'm
out
of
maces
con.
So
we
surveyed
the
landscape.
There
are
a
bunch
of
different
things
out
there
for
running
tasks
on
meso
s--,
but
we
found
that
they
were
very
generic.
A
It
wouldn't
provide
a
great
developer
experience
for
people
working
on
the
HubSpot
product,
so
we
did
something
crazy
and
we
made
our
own
meso
framework
called
singularity.
It's
still
running
to
this
day
and
it's
an
opinionated
look
at
running
Web
Services
workers
and
cron
jobs
and
meso
s--,
tailored
to
the
way
our
developers
work.
As
far
as
I
know.
At
this
time,
kubernetes
is
just
the
twinkle
in
the
eye
of
a
few
Googlers
and
it
ended
up
being
a
great
success.
We
were
able
to
save
money
by
consolidating
instances.
A
We
were
able
to
improve
reliability
by
automating
away.
All
those
3:00
a.m.
wake-up
calls
and
we
were
able
to
improve
productivity
by
making
it
easy
for
developers
to
do
things
that
mattered
most
like
tailing
logs
are
taking
heap
dumps
or
whatever
and,
as
you
can
see
by
a
younger,
less
stickered
Tom,
we
bragged
about
it
a
lot.
A
So
time
goes
on
being
a
product
developer
is
now
not
so
bad.
You
have
singularity,
but
what
about
the
infrastructure?
The
infrastructure
at
HubSpot
was
still
kind
of
in
that
twenty
thirteen
world
of
you
know
lack
of
automation.
Life
was
hard.
We
could
have
taken
the
route
of
adding
stateful
support
to
singularity,
but
we
really
liked
how
singularity
just
did
one
thing
well
and
we
didn't
want-
have
to
reinvent
a
bunch
of
different
wheels.
So
at
that
time
we
kicked
the
tires
on
kubernetes.
A
We
liked
what
we
saw
and
we
decided
that
we
would
explicitly
try
to
do
our
state
fall
workloads
on
kubernetes
and
worry
about
consolidating
singularity
and
qumran
aid
kubernetes
down
the
road.
So
all
of
our
infrastructure
teams-
data
infrastructure
was
the
first
to
take
the
plunge
and
the
hottest
fire
in
that
world
was
my
sequel
and
we
got
Alex
from
the
sequel
team
here
to
tell
more.
B
That
means
we
ran
a
lot
of
my
sequel
and
just
to
put
numbers
on
it.
I'm
talking
about
roughly
450
my
sequel
clusters
back
in
2016
and
a
cluster
here
means
generally
two
machines,
possibly
or
but
like
a
master
and
some
number
of
slaves
and
the
450
clusters
are
225
in
QA
and
225
in
prod.
So
if
you
do
the
math
there
we're
talking
about
like
at
least
900
ec2
instances,
but
actually
in
reality
closer
to
a
thousand
and
that's
painful
right,
that's
expensive
and
operationally.
B
It
doesn't
scale
right
with
more
services
coming
up
all
the
time.
We
just
have
a
worse
and
worse
problem,
because
Amazon
has
a
non
zero
impairment
rate,
which
means
somebody
on
my
team
gets
paid
and
you
put
hands
on
keyboard,
run
an
ansible
play
or
two
or
three
of
them,
so
that
sucks
and,
like
I,
said
it's
getting
worse
all
the
time,
because
we're
always
spinning
up
new
micro
services,
but
even
just
to
do
that
right.
B
Your
team,
your
product
team
files,
a
JIRA
and
then
somebody
on
my
team
has
to
provision
some
virtual
machines
with
a
puppet
class
and
set
some
puppet
variables
and
all
the
while.
Your
product
team
is
just
waiting
and
waiting
and
waiting.
The
other
axis
it
wasn't
scaling
on
was
the
actual
database
itself,
the
query
and
data
load.
You
know
with
no
real
idea
how
big
your
service
is.
Gonna
become
inside
the
HubSpot
app.
B
The
other
thing
you
could
do
is
build
sharding
into
your
application,
which
is
itself
a
nightmare
and
really
just
a
nightmare
that
you
would
ever
just
want
to
do
once
not
a
thousand
times
right.
We
have
like
1500
micro
services,
so
yeah
that's
also
pain.
Yes,
so
we
needed
something
that
would
meet
us
on
the
operational
front
and
the
data
scalability,
and
we
looked
around
so
it's
end
of
2016
at
that
point,
maybe
like
September
October
November,
and
this
is
what
we
saw
right.
B
So
here's
situs
and
cockroach
and
RDS
and
Valera
and
by
Tess,
that's
what
we
saw,
but
we
don't
really
want
to
use
like
situs
or
cockroach,
because
we
want
to
stick
with
my
sequel
right.
It's
what
we
know.
It's
what
we
have
all
the
operational
experience
with
and
we
weren't
really
comfortable
with
handing
our
infrastructure
off
to
a
third
party
like
AWS
and
pay
through
the
nose
and
deal
with
more
or
less
Glacial
support
turn
around
and
it
doesn't
actually
scale
horizontally
right.
B
It
doesn't
shard,
so
that's
kind
of
a
non-starter,
not
a
not
a
good
fit
Rob
spot
and
we
knew
like
Tom
was
saying
that
the
rest
of
our
infrastructure
will
be
moving
in
the
kubernetes
direction.
So
we
wanted
something
that
would
at
least
rhyme
with
kubernetes
and
so
Galera
extra
DB
cluster
or
whatever
that's
out,
and
so
we
landed
on
by
tests
and
we
really
liked
what
we
saw
its
my
sequel
base.
It's
open
source,
it's
cloud
native,
and
we
really
liked
that
it
wasn't
new
at
that
time.
B
Back
in
2006,
it
had
been
deployed
in
production
at
YouTube
for
several
years.
So
what's
my
test.
Hopefully,
you
got
to
go
to
one
of
the
talks
earlier
this
week.
We've
had
a
few
really
great
talks,
but
even
if
you
did
I
want
to
highlight
a
few
parts
of
the
architecture
they're
over
on
the
right,
you
see
it's
labeled
admin.
It's
the
control
plane,
there's
a
bunch
of
boxes
there,
but
the
thing
it
doesn't
say
is
that
this
vtc
TLD,
that's
the
vy
test.
Control
daemon
actually
has
an
API
that
you
can
code
to.
B
So
that's
awesome.
The
data
plane,
however,
is
consists
of
two
parts:
the
VT
gate,
which
is
a
proxy
there
and
VT
tablets,
which
are
like
sidecars
that
run
next
to
your
my
sequel
process.
Yeah.
So
that's
a
lot
of
boxes
right.
How
do
you
set
it
all
up?
Well,
it
should
run
in
kubernetes
right.
Well,
not
really.
It
was
like
designed
to
run
in
borg
should
run
kubernetes,
so
we
just
got
started
and
we
got
started
like
anybody
would
write
with
helm,
but
it
got
a
little
unwieldy
right.
B
That's
a
lot
of
different
helm
charts
and
it
didn't
really
give
us
the
control
that
we
wanted
right.
I
never
ever
want
to
roll
out
a
release
and
have
all
five
hundred
databases
see
the
numbers
gonna
go
up
as
the
time
goes
up.
All
five
hundred
databases
get
updated
right
then
that's
like
right,
I
need
Canaries.
I
need
all
that
kind
of
stuff,
so
around
let's
say
January,
maybe
of
2017.
B
Somebody
on
my
team
found
this
blog
post
from
core
OS
introducing
operators,
so
that
I
guess
was
written
November
2016,
but
we
didn't
really
have
a
need
for
it,
didn't
see
it
until
January
or
so
and
we
fell
in
love.
We
scrapped
helm
immediately
and
started
writing
an
operator,
and
this
you
know
this
is
like
well
before
operator
SDK,
so
we
didn't
have
any
of
that.
So
we're
just
writing
against
client
go
so
hopefully
you've
seen
something
about
operators.
B
B
B
You
got
a
performance
class
that
shows
like
the
size
of
resource
limits
and
requests
for
your
pod,
how
many
replicas,
and
that
last
one
they're
sensitive,
just
to
say,
like
HubSpot,
s--
public,
so
this
one
is:
does
the
database
contain
data
which
should
be
considered
sensitive
for
sarbanes-oxley
compliance?
So
it's
a
mouthful
and
our
operator
translates
that
bit
of
yeah
mole
into
three
different
things:
pod
disruption,
budget,
which
tom
will
talk
about
a
little
bit
later,
a
cron
job
that
we
use
for
backups
and
a
stateful
set
and
yeah.
B
So
I
don't
have
like
two
hours
to
talk
to
you
about
all
the
work
my
team
did,
but
I
want
to
hit
on
some
of
the
high
points
we
do
have.
Another
custom
resource
that
manages
pools
of
Vth
describes,
pools
of
ET
gates.
We
run
several
different
pools
that
you
know
pass
different
options
into
the
underlying
VT
gates
and
we
want
to
be
able
to
scale
those
things
differently.
So
that's
that
HubSpot
looked
at
the
options
for
JDBC
driver
and
while
Vai
test
speaks
the
my
sequel
wire
protocol.
B
Another
bit
of
work
we
did
was
to
integrate
Orchestrator
into
our
installation
if
you're
not
familiar
with
it.
Orchestrator
is
the
h,
a
solution
for
my
sequel
out
of
github,
great
piece
of
software
and
finally,
as
you
might
expect,
if
I
test
grew
up
at
YouTube,
but
hub
spots,
corpus
of
queries
is
different
from
the
corpus
of
queries
at
YouTube,
so
we
actually
had
to
augment
the
grammar
of
the
sequel
dialect
of
I
test,
and
we
did
that
in
a
couple
of
parts.
B
B
The
other
thing
I'd
like
to
talk
about
is
how
we
did
the
migration
itself,
just
because
it
comes
up
all
the
time
in
the
Vitis
slack
Channel.
It
was
really
important
for
us
when
we
were
preparing
for
it
to
do
some
work
in
our
back-end
framework
library
stuff,
like
that,
so
that
we
could
make
transition
one
database
at
a
time.
This
I
agree
is
pretty
simple.
It's
just
got
one
app
one
database
in
general,
we've
got
like
a
end
to
em
setup,
but
just
to
make
the
diagram
live
with
the
bull.
B
B
We
could
proceed
with
what
I
don't
know
what
you
want
to
call
it.
A
switch
over
master
slave
swap
by
test
calls
at
re-parent,
where
we
made
one
of
the
my
sequels
running
in
kubernetes,
the
master
for
the
replication
topology
and
then.
Finally,
once
we
were
satisfied,
everything
was
working
properly.
We
could
shut
down
the
ec2
hosts
and
start
saving
money.
It's
important
to
note.
This
whole
thing
has
zero
downtime
right
at
no
point
where
we
read
only
at
no
point
where
we
totally
down
the
data
was
always
there
and
the
my
sequel.
B
So
if
we
needed
to
switch
back
to
an
earlier
phase
and
not
use
by
test
for
something
it
was
all
there,
it
all
just
worked.
It's
pretty
important
that
we
had
that
we
smooth
everything
out,
because
we
had
to
do
this
hundreds
of
times
right,
like
hundreds
and
hundreds
of
times,
but
while
my
team
was
getting
by
test
to
work
for
HubSpot,
Tom's
team
was
getting
kubernetes
to
work
for
my
tests,
so
here's
Tom
with
some
of
the
issues
that
his
team
faced
thanks.
A
Alex,
so
one
of
the
early
traps
that
we
fell
into
was
neglecting
to
account
for
how
node
maintenance
could
affect
our
services.
We
figured
that
if
a
pod
could
handle
a
deployment
scale,
maybe
a
surprise
termination,
we
were
golden
and
then
we
learned
the
hard
way
about
pod
evictions
when
we
replaced
too
many
nodes
at
once
during
a
kubernetes
upgrade
and
the
scheduler
had
no
concept
of
that
three,
you
know
all
pods
and
a
staple
said
being
evicted
is
problematic,
so
when
Hanna
did
it
and
we
experienced
downtime,
so
the
fix
here
was
easy.
A
As
Alex
said
earlier,
we
updated
the
operator
to
ensure
that
every
stateful
set
had
a
corresponding
Padma
Sri
option
budget
that
allows
you
to
define
the
rate
at
which
pods
are
evicted.
If
there's
one
thing
I
tattoo
on
my
arm,
though
it
be,
that
deletion
is
not
eviction.
This
bit
us
as
recently
as
last
week
when
we
ran
a
cube,
control,
delete
pod
and
forgot
that
that
hits
the
pod
deletion
API,
not
the
pod
of
fiction.
Api,
the
pod
deletion
API
is
not
subject
to
pod
disruption.
Budgets
so
need
a
little
surprise
there.
A
Kubernetes
kubernetes
makes
it
really
easy
to
inject
config
into
your
pods
through
secrets
and
config
maps,
but
we
run
seven
different
kubernetes
clusters
across
different
environments
in
different
regions.
So
a
given
secret
in
a
kubernetes
cluster
may
not
be
the
single
source
of
truth.
Instead,
what
we
do
is
we
store
all
of
our
secrets
in
one
centralized,
vault
cluster,
and
then
we
created
an
operator
that
will
sync
these
secrets
into
kubernetes.
The
secret
updater
operator
defines
a
custom
resource
that
we
call
a
secret
template
that
lets.
A
You
define
a
secret
that
references
values
involved,
and
it
will
make
sure
that
the
generated
secret
is
always
up-to-date
with
vault.
The
other
benefit
here
is
that
if
we
experience
it's
any
downtime
of
all,
it's
no
problem
at
all.
So
because
we're
running
things
that
aren't
necessarily
kubernetes
native.
Like
my
sequel,
we
have
to
write
out
a
lot
of
configuration
files
in
our
pods
and
one
way
you
could
do
that
is
by
having
config
maps.
A
But
all
of
our
my
sequel
pods
are
slightly
different
and
that
would
result
in
a
dizzying
number
of
config
maps
and
we
don't
want
to
have
to
manage
all
that.
So
instead
we
built
a
tool
that
will
take
care
of
it.
For
us,
HS
render
template
is
a
Python
script
that
we
build
into
our
base,
docker
image
and
all
it
does
is
it
accepts
a
ginger
template
and
any
combination
of
downward
API
values,
node
and
metadata
the
ammo
files
environment
variables
and
writes
out
the
result
to
another
file.
A
This
is
typically
something
that
we
do
in
the
entry
point
of
our
docker
images.
So,
as
you
know,
staple
services
in
kubernetes
means
that
you
have
to
pay
close
attention
to
pod
lifecycle.
As
Alex
mentioned
earlier,
we
used
three
stop
hooks
in
order
to
terminate
pods
gracefully
and
we
do
it
in
two
different
locations
for
the
VT
tablet
container.
A
A
What
because
Tet
the
tablet
is
proxying
these
queries
to
the
my
sequel
database.
We
also
got
to
make
sure
that
the
my
sequel
container
only
terminates
after
the
tablet
container.
Otherwise
there
could
be
a
different
service,
so
you
have
another
pre
stop
hook
that
simply
just
polls
to
see
is
the
tablet
still
running
if
it's
still
running,
let's
not
shut
down
my
sequel.
Yet
does
this
look
familiar
to
anyone?
A
This
is
what
happens
when
you're
an
idiot
and
run
out
of
resources
in
your
cluster.
So
we
hit
this
early
on.
You
know
we
would
have
people
testing
things
and
be
like
hey.
You
know
my
paw
is
pending
like
what
does
that
mean
and
we'd
have
to
dig
through
and
we'd.
You
know
we'd
assume
like
something
was
broken
and
then
finally
just
turned
out.
A
This
was
later
fixed
in
1.8,
but
we
had
to
do
something
macgyvering
to
get
around
that
in
the
meantime
and
the
way
we
did
that
is.
We
noticed
that
when
cubit
is
mounting
a
volume,
it's
shelling
out
to
make
FS
and
mountain
in
order
to
set
everything
up.
So
we
injected
a
wrapper
onto
the
cubelet
path
that
it's
insane.
When
you're
mounting
a
volume,
it
calls
out
to
AWS
to
see
what
are
the
tags
on
this
EBS
volume
so
that
we
can
figure
out
the
name
of
the
PVC.
A
A
A
Nowadays,
you
can
do
that
pretty
easily
with
eks
or
lifts
IP,
VLAN
plugin,
but
those
weren't
options
at
the
time
what
we
ended
up
doing
was
using
the
default
docker
bridge
and
some
fancy
routes
and
some
complicated
IP
tables
rules
to
directly
bridge
pods
to
our
ec2
network.
This
was
really
cool,
but
it
came
out
of
price
because
we
were
beholden
to
the.
What
the
default
allocation
strategy
that
docker
uses
for
IP
addresses.
We
were
noticing
that
it
was
going
lowest
first,
which
is
fine,
except
for
when
you're
rolling,
all
the
pods.
A
So
we
would
see
these
issues.
Where
am
I
single
pod
would
be
torn
down
a
new
one
would
come
up
and
take
its
IP
and
there
would
be
a
period
of
time
where
that
wasn't
fully
propagated
throughout
the
cluster,
so
that
you
know
apps
would
be
trying
to
reach
a
a
pod
thinking.
It's
one
thing,
but
it
really
being
the
other.
It
wasn't
that
bad
for
my
sequel,
but
it
was
truly
bad
for
Redis,
which
at
one
point,
we
actually
had
two
clusters
join
each
other,
which
was
bonkers.
A
So
our
Vitesse
backup
strategy
involved
stopping
my
sequel
on
an
up-to-date
replica,
but
we
didn't
want
to
sacrifice
one
of
the
replicas
pause
that
we
had
for
this.
So
instead
we
used
a
cron
job
to
create
an
ephemeral
pod
whose
sole
purpose
was
just
for
backups,
so
it
would
restore
from
the
you
know
the
last
successful
backup
catch-up
on
replication,
take
the
backup
and
then
shut
down
like
anything
else
that
needs
storage
and
the
easiest
solution
was
to
simply
create
a
persistent
volume
like
all
the
others.
A
The
problem
there
is
that
these
pods
don't
last
very
long,
whereas
the
PV
sticks
around
forever.
Unless
you
intervene,
so
that's
you
know
the
ball.
The
volume
is
idle
and
money
is
wasted.
You
may
think
that
the
solution
here
is
empty
dirt
and
you'd
be
wrong,
because,
although
that
is
some
improvement,
you
still
have
to
provision
your
kubernetes
nodes
for
your
biggest
database
in
terms
of
space
and
I/o
bandwidth.
The
solution
that
we
arrived
with,
that
is
the
best
of
both
worlds,
is
using
the
Flex
volume
plugin.
A
We
created
our
own
scratch,
EBS
flex
volume
plugin
that
our
pack
of
pods
use
that
creates
an
EBS
volume
when
the
pod
starts
up
and
destroys
it
when
it
shuts
down.
So
when
there's
no
backup
running,
there's
no
volume,
no
waste,
Staples,
Center
rollouts
are
kind
of
a
crapshoot,
because
kubernetes
had
no
concept.
You
know
that
one
of
these
pause
is
a
master
or
you
know
a
more
important.
You
know
maybe
not
more
important,
but
you
know
needed
more
care
than
some
of
the
other
pod.
A
So
we
could
get
into
situations
like
this,
where
we're
rolling
out
a
change
and
we
incur
three
different
fail.
Overs,
of
course,
the
best
case
scenario
would
be
terminate
the
master
at
last.
There's
one
fail
over,
but
what
can
you
do?
Well,
we
ended
up
doing
was
taking
this
roll
out
logic
out
of
kubernetes
and
putting
it
into
our
operator
by
making
changes
to
our
stateful
sets
with
the
delete
propagation
orphan
setting
on
well.
B
So
I
think
the
main
theme
for
lessons
learned
for
us
is
that
this
was
a
transplant
operation.
My
test
grew
up
at
YouTube.
Hubspot
has
different
needs,
different
corpus
of
queries,
but
also
a
different
environment.
You
know
we're
installing
it
in
kubernetes
at
YouTube.
It
runs
in
Borg,
we
run
zookeeper
YouTube
has
whatever
it
is
chubby
you
know,
and
down
the
line.
B
So
we
have
a
real
database
as
a
service
right,
no
more
filing
a
JIRA
and
waiting
any
engineer
can
go
to
a
page,
fill
out
a
simple
form
hit
create
and
then,
like
five
minutes
later,
their
team
gets
a
little
slack
notification.
Saying
like
hey
your
database
is
ready
and
I
want
to
be
clear.
This
forum
is
dead,
simple,
it
posts
JSON
into
kubernetes.
You
know
the
watch
in
our
operator
gets
a
little
tickle
and
like
makes
pods,
and
so
we
we
do
nothing
right.
We
actually
we
get
a
slack
notification
as
well.
B
It
says
like
if
somebody's
making
a
database
yeah
yeah
yeah
high-fives
all
around,
we
have
trivial,
read
scaling
right.
If
you
want
to
add
another
pod
to
the
database,
it's
a
simple
Kubb
CTL
edit
and
it
just
happens.
We
take
advantage
of
the
kubernetes
schedulers
bin
packing
in
a
pretty
neat
way,
because
we're
saving
money.
We
actually
then
run
three
pods
for
every
database
instead
of
just
like
two
ec2
hosts
and
that
lets
us
do
a
neat
thing
because
we're
fault-tolerant
to
like
losing
a
zone.
B
In
that
point
we
can
actually
run
the
my
sequel
cluster
at
with
semi
sync
replication
and
semi
sake.
Replication
is
pretty
safe,
so
that
lets
us
turn
the
write
durability
down
into
my
sequels
and
that
improves
throughput
and,
like
is
nicer
on
your
EBS
volumes
and
probably
cheaper.
So
that's
pretty
cool.
B
We
also
have
application
agnostic,
sharding,
so
that's
great
and
then
like
zero
downtime
recharging
thanks
to
by
Tess
and
finally,
because
of
Orchestrator.
We
have
automated
impairment
handling,
which
we
actually
had
like
I,
don't
know
a
couple
months
ago.
At
some
point
we
lost
the
kubernetes
node
to
an
impairment
and
like
we
did
nothing,
my
team
did.
We
like
saw
all
the
notifications
in
our
slack
channel
from
Orchestrator,
saying,
like
Oh,
responding
to
dead
master
on
blah
blah
blah,
and
we
were
just
busy
you
like
high-fiving,
each
other.
B
It
was
awesome,
and
that
means
it
can
happen
at
night
and
I
get
to
sleep
through
it
with
no
page
I
should
have
put
like
a
big
red
line
through
that
thing.
That's
all
we
got
for
you
want
to
have
a
big
thanks
to
sugu
the
main
author
architect
of
I
test
from
YouTube
and
thanks
to
coop
con
got
some
links
there.
Thanks
everybody,
we
don't
know
the
time,
but
we
probably
have
time
for
questions.
B
B
B
Don't
know
I'm
not
like
cuz
I,
don't
use
it
right,
little
laughs,
I
check,
so
I'm
not
super
familiar,
and
probably
other
people
in
the
room
could
make
a
better
answer
to
that.
But
you
know
last
I
knew
it
didn't,
have
a
whole
lot
of
like
deploy
Canarian
right.
So
we
we
have
a
kind
of
at
this
point,
sophisticated
system
where
like.
If
we
want
an
update
across
all
of
our
databases,
we
can
designate
a
few
of
them.
B
It's
like
alpha,
QoS
and
roll
the
database
out
their
role,
the
change
out
to
that
and
then
be
able
to
like
halt
the
thing
and
roll
back
and
stuff
like
that,
and
if
it
proceeds
like
keep
going
and
then
roll
out
a
little
bit
faster.
So
we're
not,
you
know
totalling
our
thumbs
for
forever.
So
last
I
knew
it
didn't
do
that,
but.
B
Have
a
smooth,
like
you,
know,
spray
this
DDL
out
across
all
the
shards
kind
of
thing
worked
out.
Yet
we
do.
We
do
have
some
tools
that
wrap
that
essentially
wrap
PT
online
schema
change,
we're
evaluating
another
github
project
right
now
called
ghost,
so
the
github
online
schema
change
and
and
yeah
I
think
we're
just
gonna
replace
the
backing
in
our
wrapper,
like
away
from
PT
on.
A
A
A
B
So
essentially,
the
operator
gives
us
the
control
of
how
quickly
we
roll
things
out
right,
I
as
I
understand
it
like
you,
give
helm
some
updated
charts
or
so
their
new
release
or
something,
and
then
it
just
rolls
it
out
instantly
to
everything
and
like
we
don't
wanna
do
that,
and
so
we
could
have
had
like.
Oh
here's,
a
helm
release
for
like
so
now
for
QoS
things
but
yeah.
In
the
end,
we
didn't
so
it's
it's
a
little
bit
of
a
change
of
perspective,
right,
helm
and
helm.