►
Description
Shlomi Noach, Senior Infrastructure Engineer at GitHub, introduces GitHub's latest open source release: gh-ost, a schema migration solution for MySQL.
gh-ost changes the online migration paradigm and overcomes today's limitations and difficulties in MySQL table migrations. gh-ost was designed with performance and operations in mind. It is triggerless, controllable, auditable, suspendable, and testable.
Shlomi will discuss the reasoning for embarking on developing a new solution, the existing MySQL migrations landscape, gh-ost's underlying logic, operational perks, future plans and surprising implications. Shlomi will also share thoughts on the open source landscape and the role he expects gh-ost to play.
About GitHub Universe:
GitHub Universe is a two-day conference dedicated to the creativity and curiosity of the largest software community in the world. Sessions cover topics from team culture to open source software across industries and technologies.
For more information on GitHub Universe, check the website:
https://githubuniverse.com
A
Welcome
everyone.
Thank
you
for
coming.
My
name
is
Serena
I'm
with
the
database
infrastructure
team
at
github.
This
is
a
technical
talk
about
ghost.
Our
online
schema
migration
solution
for
MySQL
do
stick
around,
even
if
you're,
not
very
technical.
So
this
this
talk
is
gonna,
be
to
some
extent
technical
and,
to
some
extent
talk
about
flow
in
operations
and
expectations
of
engineers
and
database.
A
Administrators
I
am
an
engineer
also
DBA
a
working
with
my
skill
for
15
years
now,
I'm
active
in
the
open-source
community
I
also
a
bunch
of
open
source
projects,
I'm
happy
to
work
with
a
great
team
at
the
database
infrastructure
group
at
github
and
with
excellent
engineers
who
make
it
easy
for
us
to
to
do
our
thing.
So
the
TLDR
ghost
is
our
MySQL
senior
migration
tool
from
high
school.
We
use
it
in
production,
we
altered
it
and
we
open
sourced
it.
Thank
you
for
coming.
It's
great
to
have
you
around.
A
I
would
like
to,
however,
drill
down
a
little
bit
and
explain
the
problem.
What
the
existing
environment
is
why
we
chose
to
embark
on
a
new
solution
and
how
it
went
for
us,
the
TLDR.
Exactly
all
right,
so
my
skill
get
up
is
about
git
right.
We
host
gets
git
repositories.
We
have
a
great
talk
later
on
about
spokes
our
resilient
get
set
up,
but
all
the
metadata
of
get
up
lies
within
my
skills.
So
that
means
all
your
comments,
issues
pull
requests
users
or
it's
whatever
reactions.
Everything
lies
within
MySQL.
A
My
skill
is
the
dragon
power
between
the
behind
github.com
right.
My
skill
is
down
that's
very
bad
news
for
us.
We
really
need
my
stuff
to
be
highly
available
and
highly
responsive.
We
want
right
to
have
low
latency.
We
want
to
have
high
throughput
for
right.
Very
importantly,
we
want
so
we
use
my
cell
replication
master
replicas.
We
really
want
to
keep
our
replication
lag,
very
low,
we're
going
to
talk
about
that
a
little
bit
later,
so
that
those
are
our
expectations
of
my
skill
and
my
skill.
Migrations
are
a
a
known
thing.
A
The
built
in
solution
for
my
skill,
schema
migration
is
unsatisfactory
and
I
could
drill
down
into
that
for
a
full
hour.
But
basically
it's
it's
inconsiderate
of
the
master
load.
It's
inconsiderate
of
replication
lag
and
there
have
been
tooling
around
since
2009
to
simulate
or
emulate
a
migration
and
I'm
going
to
describe
that
shortly
and
most
notable
of
these
are
PD
own
schema
change
by
procurement.
Okay,
raise
your
hand
if
you
is
peeling
on
schema
change.
For
my
skill,
migration,
that's
cool,
keep
raise
your
hand
if
you
use
the
Facebook
solution.
The
face.
A
We
are
growing
rapidly
in
traffic
in
volume
and
in
number
of
Engineers,
and
we
as
the
database
infrastructure
team,
need
to
deliver
right.
Engineers
need
to
be
able
to
do
schema
changes,
so
our
CEO
Chris
told
you
this
morning
about
new
improvements.
We
made
code
reviews,
pull
request,
reviews,
etc.
These
all
require
a
load
of
schema
changes.
We
do
them
daily.
We
sometimes
have
multiple
schema
changes
per
day.
Today
is
a
quiet
day.
A
We're
not
going
to
do
anything
but
basically
we're
doing
a
load
of
migrations
all
the
time,
not
everything
it
is
user
facing,
but
our
engineers
work
rapidly
and
need
to
deliver
on
a
base
daily
basis
and
we,
as
the
database
infrastructure
team,
certainly
don't
want
to
be
the
blockers
for
that.
So
we
are
responsible
to
make
sure
migrations
work
and
if
they
don't
that's
very
bad
news
for
us.
So
in
fact,
we've
had
migration
pains
so
late,
2015
early
2016,
we've
start
to
experience
a
lot
of
problems
with
my
scream.
A
Immigrations
regressions
were
just
run
in
too
slow
and
we're
creating
too
much
of
overhead
and
overload.
So
some
aggressions
will
just
cause
the
master
to
go,
the
muscular
master
to
go
slower
and
right.
Latency
and
increase
replication
lags
and
people
would
notice.
Yeah
good
of
that
calm
is
like
yeah
and
they
would
tweet
about
it.
And
then
we
said:
okay,
some
aggressions
are
too
heavy
wait.
We
should
move
them
to
off-peak
hours
right.
A
We
would
run
them
during
the
night,
but
some
of
them
were
so
long
that
we
would
just
shift
them
to
the
weekend,
and
then
they
were
still
causing
so
much
pain
that
we
would
need
actually
babysit
them.
We
would
have
one
or
two
people
taking
watches
and
sitting
nearby
the
computer
just
to
make
sure
that
they're
running
well
and
they
weren't.
We
had
some
aggression
causing
outages
to
github.com.
Some
some
of
the
outages
in
2016
were
a
direct
result
of
migrations,
gone
bad.
In
fact,
we
had
this
notorious
table.
A
It
was
it's
like
our
nemesis
or
our
ik
enemy
table
which
consistently
crashed
our
server
our
master
during
migration.
We
couldn't
migrate
that
table
so
that
really
we
reached
a
point
where
this
is
not
working
out
and
we
would
push
back
on
our
engineers
who
want
to
do
the
schema
migration,
so
we
would
say:
oh
no.
This
is
a
risky
migration.
A
In
February
database
infrastructure
team
met
with
some
other
engineers
heavy
who
here
has
been
to
receive
stock
previous
slot,
not
too
many
too
bad
so
receive
is
a
platform
engineering
manager
here
get
up
and
she
was
telling
us
about
the
pains
of
migrations
the
engineers
are
experiencing
and
we
were
sitting
and
trying
to
figure
out
all
right.
So
what's
wrong
with
the
existing
solutions,
we
are.
We
run
in
periodic
schema
change
for
years
now
and
we're
trying
to
figure
out
okay.
A
Why
is
it
going
so
bad
so
before
explaining
what
what's
wrong
or
why
are
they
not
working?
Well
for
us,
the
existing
migration
tools.
I
want
to
do
a
quick
technical
dive
into
how
existing
honesty
of
immigration
tools
work,
I'm,
going
to
review
I'm
going
to
review
pity
on
my
schema
change
and
the
Facebook
solution.
So
basically,
all
the
migration
tools
work
the
same
way.
We
have
the
original
table.
We
want
to
migrate,
there's
millions
and
millions
of
rows
in
that
table.
We
can't
just
out
the
table
because
it's
a
blocking
operation
and
yada
Arab.
A
A
Meanwhile,
intercepting
ongoing
changes
to
the
original
table
and
applying
them
on
the
ghost
table
at
Sanchi.
It's
some
stage.
We
are
convinced
that
the
tables
are
in
sync.
We
do
a
quick
stop.
The
world
thing
we
flip
the
table.
We
put
the
ghost
table
in
place
of
the
original
table
and
migration
is
complete.
That's
the
general
workflow
of
all
schema
migration
tools.
Alright,
so
far
cool
the
pre-cana
toolkit,
the
one
most
of
the
world
is
using
right
now
uses
a
trigger
based
synchronous
approach.
What
does
that
mean?
A
So
there
is
if
you're,
not
technical,
please
do
stick
around
because
there's
just
interesting
implications
after
that
there
are
not
very
deeply
technicals.
So
triggers
are
entities
in
my
skill,
they'd,
say:
okay,
if
someone
is
updating
a
row
on
that
table,
here's
what
I'm
going
to
do
and
you
would
create
your
own
schema
change-
create
three
triggers
one
to
handle.
Instance.
Segments
want
to
handle
the
lead
statements
want
to
handle,
update
statements
and
those
triggers
would
would
say:
okay,
there's
an
update
on
that
table.
A
Let's
go
ahead
and
update
the
corresponding
row
on
the
ghost
table.
There's
an
insert
on
that
table.
Let's
insert
ago
a
row
on
the
girls
table,
there's
a
lead.
Let's
delete
that
role
from
a
ghost
right.
That's
how
those
triggers
use
a
one-to-one,
simple
mapping
of
actions
running
on
our
original
table.
Mapping
them
onto
the
girls
table
the
Facebook
tool
uses
a
trigger
based
an
asynchronous
approach.
A
So
if
I'm
updating
a
role
on
the
original
table,
there's
a
new
row
added
to
the
change
log
set
table,
saying:
oh,
hey,
I'm,
updating
that
row
with
those
values
on
that
table,
so
this
table
keeps
growing
and
growing
as
migration
goes
and
then
in
a
synchronous
process,
iterates
those
that
table
and
DQ
in
the
elements
and
replaying
them
on
the
ghost
table.
So
it's
a
synchronous
approach
because
the
rows
are
being
applied
to
the
ghost
table
way
after
they've
been
applied
to
the
original
table
so
far,
so
good,
all
right.
A
So
what's
wrong
with
those
solutions
we
said:
I,
don't
really
try
to
figure
out.
Why
is
everything
is
so
painful
and
one
of
the
pretty
obvious
conclusions
we
we
came
to
is
that
it's
the
figures
and
here's
what's
wrong?
What's
wrong
with
triggers
in
my
skill
in
general
and
in
trigger-based,
my
film
migrations
in
particular,
so
triggers
in
MySQL
are
stored.
Routines
in
my
skills,
stored
routines
are
not
compiled.
They're,
not
even
precompiled
they're,
just
being
interpreted
triggers
run
from
the
same
transaction
space
where
your
queries
execute
on
right.
A
So
if
you're
issuing
an
insert
on
a
table
and
there's
a
trigger
it
hacks
or
it
hijacks
your
in
transaction
space
and
then
goes
on
to
interpret
to
do
it,
a
text
parser
full-blown
text
parser,
you
know
what
what
what
is
your
action
or
you
want
to
insert
a
corresponding
role
on
that
table
and
then
proceed
to
interpret
that.
So
that's
a
major
slowdown
to
your.
You
know
fast
going
analytic
LTP
table,
so
that's
one.
A
The
second
thing
is
looks
so
say
you
have
50
concurrent
connections
right
into
your
table,
right
they're
competing
for
some
logs
and
then
come
those
triggers
and
apply
those
changes
to
yet
another
table
independently
competing
for
other
logs
on
that
table
and
that
way
of
independently
competing
for
yet
other
logs
create
a
mayhem
of
logs,
which
is
what
typically
brings
like
service
to
a
grinding
halt.
When
migrations
go
bad.
A
A
Trigger
base
migrations
are
unsustainable,
so
the
picanha
toolkit
solution
has
a
nice
feature
of
throttling
saying
alright
I
migrated
in
table.
It's
gonna
take
12
hours
because
it's
a
huge
table
and
the
load
on
the
master
seems
high.
So
hey,
why
don't
I
step
down?
And
you
know,
wait
till
things
relax
and
then
I'll
continue,
but
it
does
that,
but
it
cannot
really
suspend
operation.
A
What
it
can
only
do
is
suspend
the
copying
of
millions
of
rows
right
the
backfield,
but
it
cannot
really
stop
the
triggers,
because
the
existence
of
triggers
is
essential
to
the
correctness
of
the
algorithm
if
the
triggers
are
dropped
or
something
that
that's
it.
That's
data
corruption,
so
the
triggers
must
keep
on
running,
and
we
notice-
and
we've
noticed
this
in
other
companies
as
well
in
our
experience
that
on
very
busy
tables
triggers
themselves,
are
enough
to
to
cause
a
major
impact
down
to
a
grinding
halt
on
your
service.
A
We
had
the
intention
of
running
concurrent
migrations,
no
trusting
triggers
for
that.
We
just
couldn't
bear
the
thought
of
having
12
15
20
triggers
all
competing
for,
looks
all
around
our
database
and
there
was
no
reliable
testing
in
the
sense
of
alright
I'm
thinking
about
writing
running
this
migration.
Is
there
way
for
me
to
know
in
advance
that
it's
going
to
cause
pains
there
isn't?
There
is
a
way
to
run
trigger-based
migration
on
replicas,
but
it
doesn't
emulate
at
all
the
same
kind
of
workload.
It
would
cause
on
the
master.
So.
A
A
Let
me
take
you
back
for
a
moment
to
the
Facebook
solution.
So
recap:
Facebook
use
triggers
to
write
the
ongoing
changes
into
a
change
log
table.
So
it
turns
out
you
don't
really
need
the
change
log
table
and
if
you
don't
leave
the
change
log
table,
that
means
you
don't
need
triggers,
because
the
triggers
are
only
there
to
populate
the
change
log
table
and
the
reason
you
don't
need
change
log
table
is
that
my
school
has
a
built-in
mechanism.
That
is
the
change
rope
table.
A
Anyone
knows
what
that
mechanism
is
I
just
said
it
yeah
the
binary
log.
Thank
you.
So
the
binary
logs
are
the
very
heart
of
the
replication
mechanism
and
everyone
has
by
naruse.
We
wouldn't
think
of
running
our
servers
without
binary
rows,
because
we
want
to
have
replication
and
that
step
number
one
to
have
a
replication.
The
binary
laws
are
themselves
change.
Logs
of
anything,
the
changes
on
the
server
that
includes
creations
of
schemas
creations
of
users
and,
of
course,
changes
inserts
rights,
deletes,
etc.
A
So
ghost
takes
an
approach
of
saying
no
I'm
not
going
to
look
for
triggers
I'm,
not
I'm,
not
going
to
listen
on
the
table
itself,
I'm
going
to
hook
out
on
the
binary
log.
In
fact,
I'm
going
to
impersonate
as
a
replica
to
the
servants
say:
oh
hey,
I'm,
a
replica,
please
give
me
binary
logs
and
the
service
sends
all
be
log
entries
and
we're
skipping
most
entries,
because
we're
not
interested
in
changes
to
other
tables
were
not
interested
in
creationist
abusers.
A
We
are
filtering
only
those
statements
that
apply
to
our
original
table
and
then
we
take
those
and
rebuild
an
SQL
query
that
replays
that
change
on
to
the
ghost
table.
Okay,
so
far,
so
that's
that's
fairly
similar
to
the
Facebook
solution,
in
the
sense
that
it's
an
asynchronous
approach
right
by
the
time
we're
reading
from
the
binary
log
that
the
data
is
long
since
being
committed
to
the
server,
but
we've
just
decoupled
the
events
from
the
table
itself
or
from
the
master
we
still
copy.
A
A
We
can
hook
up
to
a
replica
and
say:
oh,
hey,
dear
Africa,
can
you
please
ship
me
those
binary
logs,
and
that
is
actually
what
we
do
and
there's
implications,
because
that
that
gives
us
power
to
use
statement
based
replication
when
ghost
expects
a
robust
replication,
all
sorts
of
stuff,
but
the
nice
thing
is
while
colors
are
not
showing
here.
Oh
my
god,
I.
A
Alright,
so
ghost
is
actually
pulling
the
binary
logs
from
a
replica
of
floating
the
master,
the
master,
the
ghost
doesn't
talk
to
the
master
in
terms
of
binary
rows.
It
just
writes
to
the
master.
Is
that
okay,
so
far,
good,
so
ghost
is
copying
millions
of
rows
from
the
original
table
to
the
ghost
table
and
is
reading
binary
logs
from
the
replicas
to
apply
them
back
on
the
master.
A
Here
are
some
nice
consequences
to
that
approach,
and
it's
just
going
to
get
better
and
better
so
that
approach
says:
okay,
ghost
is
in
complete
control
of
all
the
rights.
All
the
rights
are
coming
from
ghost,
not
from
internal
triggers.
From
ghost
that
means
ghost
can
throttle
the
entire
operation.
If
it
wants
to
right
ghosts,
decide
I,
don't
want
to
write
to
the
master,
the
master
doesn't
see
anything.
What
does
the
master
see?
Musta
sees
a
connection
coming
from
girls
during
issue
the
inserts
and
deletes
and
updates.
A
It
doesn't
know
anything
about
a
migration
running
right.
We
have
decoupled
the
workload
on
the
master
from
the
migration
workload
in
the
trigger
based
solutions.
If
you
have
50
concurrent
connections,
writing
on
to
your
table,
the
triggers
will
make
sure
you
will
have
50
concurrent
connections.
Writing
on
to
the
ghost
table
or
the
changelog
table
right,
the
higher
the
load
of
the
master.
They
hire
the
load
of
the
migration
itself,
its
explosion
right
things
can
only
go
worse
from
that.
A
But
here
we
read
the
binary
logs
I,
don't
care
what
the
workload
was
in
the
master
by
the
time.
I
get
I,
intercept
those
queries,
they're
streamlined,
they're,
sequential
I,
don't
know
if
it
was
50,
concurrent
connections
or
one
concurrent
connection,
I,
don't
care
we're
right
in
sequentially.
In
fact,
ghosts
also
see
realizes
the
row
copy,
along
with
the
beam
log
events
applying.
So
that
ghost
says:
okay,
let's
copy
a
thousand
rows.
Oh
wait
now.
A
I
have
a
few
big
log
events:
let's
apply
them,
let's
copy
the
next
1000
rows,
so
everything
is
sequential,
which
means
all
the
master
sees
is
a
single
connection.
Writing
sequentially,
which
means
there's
no
lock
contention
at
all.
It's
not
that
that
we've
decoupled
the
right
contention
on
the
original
table
from
the
girls
table.
There
is
no
contention
on
the
ghosts
table,
everything
is
serialized
and
my
skill
really
likes
it
when
you
don't
do
contention,
if
you're
good,
to
write
sequentially
what
my
skill
is
very
happy
about
it.
A
So
it
gives
us
a
few
options
like
this
decoupling
means
you
know
what
I
can
work
directly
on
the
master,
pulling
the
binary
logs
from
the
master
and
applying
them
or
I
can
do
what
I
just
explained,
I'm
pulling
the
binary
logs
from
a
replica
and
then
riding
on
to
the
master
or
I
could
choose
option
number
3
and
just
migrate.
The
table
on
the
replica,
so
I
can
completely
ignore
the
master.
Why
do
I
care?
It's?
A
The
same,
binary
looks
here
and
here
and
it's
the
same
table,
so
the
algorithm
is,
is
the
same
in
all
three
options.
It's
just
a
matter
of
where
you
point
your
writes
and
reads
so
we
can
migrate
tables
on
a
vertical.
Why
would
we
want
to
do
that?
Because
that
gives
us
powerful
testing
which
lead
us
leads
us
to
the
next
issue
of
trust,
so
we've
been
using
periodic
schema
change
for
years.
The
world
is
using
people,
honest,
kimchi,
schema
change
for
use.
The
world
is
finding
issues
reporting
issues
this.
A
This
has
been
polished
for
years.
It's
it's
a
well
known
solution,
it's
not
working
for
us,
but
it's
trusted
its
trusted
with
the
data.
It's
not
going
to
corrupt
our
data
or
something-
and
here
we
are
offering.
Oh
hey,
we
conceived
this
in
February
and
we
want
to
run
this
right
now
on
get
up,
and
this
is
their
github
data
right.
A
If
we
don't
do
this
right,
very
bad
news
right,
we
have
backups
here
we
have
solutions,
but
no
way
I
mean
if
we're
going
to
even
think
that
some
tiny
bit
is
going
to
get
corrupted
because
something
is
going
wrong.
No
way,
this
is
not
going
to
go.
So
what
gives
us
this
trust
that
a
new
tool
we've
developed
in-house
and
we're
the
first
to
test
it
right?
What
gives
us
these
trusted?
This
is
working
out.
How
can
we
put
that
to
production?
A
So
ghost
goes
a
long
way
to
build
that
trust,
and
the
first
is
of
course,
testing
and
other
than
an
integration
testing
unit
areas
which
is
nice,
but
you
never
know
in
production.
Ghost.
Has
this
built-in
feature
to
do
a
test
migration
in
production
that
really
migrates
your
production
data
without
affecting
production?
So
that's
that
we
call
that
feature.
That's
a
test
on
replicas.
How
does
that
work?
A
We
connect
to
a
replica
and
we
listen
on
the
binary
extreme
on
the
replicas
and
we
apply
the
changes
back
on
the
replicas
right.
So
it's
a
real
production
workload
and
we
are
really
copying
millions
and
tens
of
millions
and
hundreds
of
millions
of
rows,
just
as
we
would
on
the
master
and
it
mimics
to
reasonable
extent
the
kind
of
workload
it
would
on
a
master,
but
more
on
this
later
and
then
at
the
very
end
we
don't
actually
flip
the
tables
we
stop
replication.
A
We
make
sure
the
tables
are
in
sync
as
far
as
ghost
is
concerned,
and
we
keep
replication
stopped
at
which
time
you
have
the
original
table
and
the
ghost
table
both
in
place
supposedly
sync,
supposedly
because
now
it's
your
turn
to
look
at
the
data
here.
Look
at
the
data
here.
Compare.
Are
you
satisfied?
Yes,
Trust
is
built.
In
fact,
this
is
how
we
test
goes
in
production,
so
we
have
master,
we
have
production
replicas.
A
Then
we
have
special
testing
replicas
for
ghosts
and
each
and
every
one
of
those
replicas
is
doing
a
continuous
full
coverage
of
all
our
production
tables.
One
at
a
time
we
migrate
a
table
on
that
replica.
Real
production
table
real
production
traffic
any
hour
of
the
day.
It's
such
test
results
with
both
tables
in
place.
We
do
a
full
Select,
like
a
full
dump
of
both
those
tables.
We
check
some
the
checks.
Summers
are
identical,
we're
happy,
not
identical.
Red,
Alert,
red
alert,
bad
right.
We
anytime,
we
are
releasing
a
new
version.
A
We
put
that
to
testing
a
couple
of
days.
These
three
servers
are
hammering
everything
we
tested
like
a
few
thousands
times
our
production
tables
before
running
our
first
production
migration
on
the
master,
so
we
already
had
some
trust
being
built
by
now,
we've
probably
ran
dozens
of
thousands.
I
need
to
check
migrations,
and
this
will
be
in
the
hundreds
of
thousands
and
well
migrations.
Take
time
I
don't
know
when
we're
going
to
reach
millions,
but
we
know
that
any
of
our
production
table
has
been
migrated
with
a
hundred
percent
success
ratio.
A
So
that
gives
us
confidence
so
that
builds
our
confidence
in
the
correctness
of
the
data.
How
about
the
the
load
and
the
operational
aspects
so
I
already
mentioned
ghost
knows
how
to
throttle
but
really
throw
those
such
that
if
I
say,
hey,
goes
stop
what
you're
doing
the
master
is
just
back
to
normal.
Now
we've
made
this
accessible
I'm
going
to
show
you
how
a
bit
later
through
chat
ups.
Anyone
can
connect
to
ghost
and
say
yeah.
You
know
what
I'm
not
feeling
good
about
it
and
by
the
way
we
don't
do
that.
A
But
we
can,
we
can
say:
hey
I,
don't
feel
good
about
it.
Let's
throttle
the
operation
and
then
go
says
alright.
I'm
not
doing
anything.
From
now
on,
your
master
is
back
to
normal
I'm
gonna
just
sit
her
and
wait
and
wait,
and
this
can
go
on
for
seconds
minutes
or
hours
or
even
more
than
a
day.
I
wouldn't
want
to
suspend
more
than
the
day,
but
this
could
go
for
more
than
a
day,
and
so
we
have
this
safety
net
of
saying.
This
is
not
going
right.
A
I'm
gonna
throw
this
now
we're
today
at
the
stage
where
we
don't
do
that,
because
we
do
trust
the
ghost
operation,
but
knowing
that
we
can
do
that
gives
us
a
lot
of
trust
into
ghosts.
We
can
always
say:
please
stop
car
over
category
is
the
most
scary
part
or
the
more
the
riskiest
part
of
the
operation.
A
That's
where
the
tables
are
being
flipped,
it's
riskier
because
there's
metadata
logs
involved-
and
there
is
stop
the
world
kind
of
thingy
that
you
need
to
flip
the
tables
before
rights
can
resume
and
most
DBAs,
I
know,
have
the
tendency
to
be
present
at
the
computer
at
the
time
of
the
car
over
right.
In
fact,
if
ABA
knows
that
migration
is
known
to
run
for
like
12
hour
ish,
and
it's
now
12
p.m.
I'm
not
going
to
kick
this
migration,
because
I'm
gonna
have
to
be
awake
at
midnight
to
watch
over
it.
A
No
I'd
better
log
in
at
9
p.m.
from
home
and
then
run
it
so
that
in
9
a.m.
I'm
at
the
office
ready
to
observe
what
happens
at
the
car
over
time
and
that's
craziness
right.
But
why
would
I
change
my
personal
schedule
to
fit
what
it's
it's
a
script
come
on.
Iii
shouldn't
change
my
life
because
of
a
script.
So
ghost
goes
your
way
and
says
you
know
what,
when
I,
when
I'm
done,
copying
those
millions
and
millions
of
rows,
and
given
that
you
told
me
you
want
to
because
I
don't
have
to.
A
But
if
you
configure
me
to
do
that
when
I'm
done
copying,
the
millions
of
rows
and
I'm
ready
to
cut
over
I,
don't
really
have
to
cut
over
hey
I
can
continue
running
for
a
while
syncing
those
tables
I'm
just
intercepting
more
being
log
events
and
applying
them.
I
can
continue
that
for
a
few
seconds
or
minutes
or
hours.
A
In
fact,
I
will
never
cut
over
until
they
say
until
you
say
so
so
we
kick
them
aggression
whenever
we
feel
like
kicking
an
aggression,
starting
immigration,
and
we
know
that
it's
not
going
to
cut
over
unless
we
say
so
so
IKEA
murasia.
Is
it
going
to
complete
the
3
a.m.
I?
Don't
care
I'm
gonna
wake
up
tomorrow
morning
and
ghost
is
going
to
tell
me.
Oh
yeah,
you
know
I've
been
ready
for
you
for
four
hours,
but
the
table's
already
I'm
I'm
still
thinking
them.
A
Whenever
you
say
the
word
and
then
we
say
yeah
ghost
please
cut
over
and
then
I'm
there
physically,
but
the
machines
waited
for
me.
I
didn't
wait
need
to
wait
on
the
machine
right.
That
is
a
that
is
a
serious
shift
of
paradigm.
In
my
personal
life
view
of
how
things
should
be
done
right,
the
machines
should
adapt
to
my
lifestyle,
not
me
to
theirs
right,
so
the
trust
means
I
can
go
to
bed,
I
can
go
out,
I
can
go
to
dinner
and
not
being
old.
Immigration
is
about
to
commit
no
I.
A
Don't
care,
it's
going
to
wait
for
me.
Ghost
can
be
configured
to
be
nice
to
sleep
more,
which
means
reduce
load
and
here's
the
this
is
like
the
latest
and
greatest
in
you.
I'm
gonna
show
you
a
graph
in
well,
so
we
are
able
to
measure
replication
lag
with
a
sub
second
resolution
right
typically
at
github
we
have
sub
second
replication
lag
throughout
the
day.
Normally
we
have
like
way
below
one.
A
Second,
I'm
gonna
show
you
a
graph
if
replication
lag
is
like
five
seconds
we
were
like
what's
going
on
and
if
it's
like
I,
don't
know
30
40
seconds.
This
is
like
a
call
for
investigation.
Why
would
replication
lag?
Go
beyond
30
40
seconds?
For
us,
we
don't
like
it.
Replication
lag
means
the
replicas
are
stale,
which
means
someone
at
github.com
is
going
to
see
data
that
is
30
seconds
old,
which
means
if
they
just
committed
something.
A
If
they
just
commented,
they
might
not
see
that
change
coming
and
it's
like
what's
going
on
at
github.com,
so
we
really
want
to
keep
the
replication
like
small,
so
gross
disabled,
so
ghost
monitors
are
replicas
lag
and
pitiless
schema
change.
Also.
Does
that
right
and
whenever
they're
replicas
lag,
we
throttle
sorry
the
operation,
but
we
were
able
to
do
that
based
on
a
sub
second
resolution.
So
here
goes.
This
is
gonna.
Tell
your
story
in
a
while.
This
is
our
very
first
sub.
A
A
This
is
about
point
one
point
two
seconds
which
were
good
with
I'm
happy
to
have
this,
and
here
is
where
I
kicked
a
migration
and
I
couldn't
the
migration
to
throttle
at
500
milliseconds
lag,
and
what
you
can
see
is
that
throughout
and
40
minutes
or
so
it
was
running
with
replicas
like
you
know
the
Purple
Line,
it's
actually
it
shouldn't
be
there,
but
all
the
rest
line
of
the
lines
are
well
below
1
second
lag
right
and
then
40
minutes
into
that
migration.
So
well
because
that's
the
first
time
I
did
sub-second
migration.
A
A
You
remember
our
nemesis
or
arc
enemy
that
table
we
couldn't
migrate.
That's
the
one
where
we
said.
Alright,
we're
running
this
prime
time.
We're
running
subsection
migration
lag
and
I'm
running
I'm
working
remotely
I'm
based
in
Amsterdam
I,
go
to
the
Armstrong
office,
but
usually
I
work
from
home.
My
wife
she's
also
an
engineer
she
works
at
reddit.
She
also
works
remotely,
so
we
share
the
same
office
space
and
we
were
actually
able
to
talk
business.
She
understands
my
work.
I
hope
to
understand
hers
and
we
were
sitting
and
I
was
I.
A
Was
writing
this
and
this,
like
our
killer,
table
right
our
nemesis,
and
this
is
the
first
time
I'm
running
sub-second
transition
I'm.
Looking
at
the
graph
easy,
oh
honey,
she's
like
what
oh
honey?
What
honey?
Would
you
look
at
those
brats?
Would
you
please
look
at
those
graphs?
I
was
like
this
graph
I'm,
sorry
for
being
superlative,
but
this
graph
is
what
makes
us
so
happy
with
ghosts,
because
it
moves
that
Brutus
from
there
impossible.
This
table
cannot
be
migrated
into
the
unnoticeable.
We
are
now
able
to
migrate
any
table
any
time
of
the
day.
A
We
don't
care.
It's
it's,
prime
time.
It's
it's
Russia.
We
don't
care.
Migrations,
are
going
to
run
and
they're
going
to
go
way
below
the
radar
and
nobody's
covering
it.
No
we're
running
that
migration.
Now
nobody
cares.
So
this
is
where
ghost
has
brought
us
from
the
impossible
to
the
unnoticeable,
which
is
why
we're
so
happy
about
it.
A
Furthermore,
ghost
gives
us
dynamic
visibility
and
control,
which
means
with
bidding
on
schema
change.
You
would
run
immigration
and
you
said
chunk
size
equals
mm.
Oh
my
god,
three
hours
into
the
migration
and
this
is
causing
too
much
load.
Let's
terminate
a
migration
reconfigure
run
it
again.
No,
no!
You
just
wasted
three
hours
of
heavy
workload
on
your
service.
No
ghost
listens
on
TCP
and
and
UNIX
circuit.
You
can
connect
to
it
and
reconfigure
on
the
fly.
Hey.
A
You
want
to
throw
a
little
thrall
now,
hey,
you
want
to
change
max
like
Minnie's
change
max
like
minutes,
so
we
can.
We
have
this
assurance
that
we
can
always
control
a
random
migration.
We
like
to
work
with
chat
ups,
everything
anyone
who's
been
to
Rossio,
stop
I've
already
seen
this.
We
share
our
work
through
chat
and
we
use
our
friend,
you
built
a
robot
to
control
things
to
do
things
for
us
and
we're
now.
We've
now
integrated
ghost,
there's
still
ongoing
work
right.
A
It's
it's
relatively
new
to
us
and
we've
integrated
parts
of
ghost
into
you,
but
such
that
we
can
say:
oh
hey
migration,
please
cut
over
or
please
reconfigure
itself
or
hey.
What's
the
status
of
a
running
migration
and
ghost
talks
to
us
through
chat
and
we
control
goes
through
chat,
and
it
looks
like
that.
Oh
my
god,
you
can't
see
anything,
but
so
sorry,
so
imagine
ghost
would
tell
you.
Oh
if
you
stand
really
close,
you
can
see
so
everyone
come
over
all
right.
Girls
tell
me:
oh
hey,
I
am
migrate
in
that
table.
A
This
is
migration
has
been
going
for
ten
hours
and
I'm
at
ninety
nine
point
three
percent,
and
then
it
says:
oh
hey,
you
know
what
I
just
completed
the
copy
and
I'm
ready
to
cut
over,
but
I
was
configured
to
postpone
the
car
over,
because
someone
went
to
yoga
class,
Jonah
and
ping
Swami
because
he
issued
immigration.
So
why
didn't
you
cut
over
and
I'm
like
okay
migration?
Please
cut
over
and
go
stairs
all
right.
Then
I'm
done
I
flipped
the
table.
A
A
We
have
just
decoupled
migrations
from
the
data
base
team.
Anyone
can
go
into
Chan
and
see
what's
going
on
with
migrations.
He's
Ross
he's
not
a
MySQL
engineer
at
github,
but
he
can
go
into
the
databases,
room
of
the
data
base,
ops,
room
and
say:
okay,
what's
up
with
migrations,
you
know
what
it's
reaiiy,
AM
and
I'm
I'm,
awake
and
I.
Think
something's
not
wrong.
I
can
struggle
if
I
want
to,
or
I
can
cut
over
for
you
if
I
want
to
or
I
can
check
what
the
status
is.
A
This
is
how
share
ups
is
all
about.
Likewise,
I
can
do
things
that
Ross
is
responsible
for,
but
I
have
visibility
into
into
what's
going
on,
so
everyone
has
trust
into
the
operation,
not
because
some
DBA
magic
is
going
on,
but
because
everyone
can
see
that
and
that
that
is,
that
is
really.
That
is
really
awesome.
So
Wow
no
I
really
have
to
tell
you
about
that.
So
here's
a
look
into
a
possible
future.
A
Basically,
the
future
says
we
just
want
everything
to
be
stable
and
good,
and
have
the
world
adopt
ghosts
ghosts
as
opposed
to
running
existing
tools.
Here's
one
idea:
we
have
sort
of
our
immediate
roadmap,
but
it's
like
we
have
to
have
this
so
remember
how
I
said:
we're
still
copying
rows
like
meals,
the
meals
rows
from
the
original
table
to
the
ghost
table
right,
there's
still
a
coupling
of
the
two
tables
in
here
right.
A
We
could
be
couple
it
by
reading
the
rows
from
the
original
table
into
ghost
and
then
write
them
back
to
the
ghost
table
and
in
this
way,
we'll
we're
decoupling
the
the
copy
of
rows
right,
we're
more
in
control.
We
means
we
can
use
a
replica
and
read
the
rose
from
the
original
table
on
the
replica
and
then
only
write
them
back
to
the
ghost
table
on
the
master,
which
means
the
master
now
truly
and
utterly
doesn't
know
anything
about
immigration.
A
A
It
doesn't
mind
and
that's
the
greatest
thing,
because
I
have
this
hose,
that
I
can
point
it
into
the
master
and
say
yeah
I'm
gonna
write
into
that
table
and
the
master
doesn't
care,
and
if
the
master
doesn't
care,
then
no
one
cares,
which
means
we
can
point
those
rights
in
to
another
master.
That
is
completely
unrelated
to
our
topology,
and
that
means
we
can
do
live
table
migrations
cross
clusters.
We
can
say:
oh
yeah,
you
know
that
table
is
too
big
for
our
service.
We
want
it
elsewhere.
A
Oh
sure,
let's
run
ghost,
it
will
copy
the
table
into
another
server
and
streamline
it
and
intercept
changes
until
it
says
it's
ready
and
then
I
could
cut
over
or
I
could
wait
until
I
really
want
to,
and
then
it's
some
stage
kill
the
app
connections
on
that
server,
kill
the
table
here
and
just
repoint
the
app
into
another
server
in
there
with
migrated.
Our
our
data
live
into
another
server.
Alright,
this
is
the
queue
for.
Yes,
it
did
you
all
right,
come
on
you're,
not
doing
your
job.
A
A
We
released
it
to
the
open
source
only
after
we
started
actually
running
migrations
in
our
production,
because
we
know
giddap
has
like
good
traction
and
we
wanted
to
feel
very
confident
about
this
tool
before
releasing
it
to
the
open
source
and
that's
less
someone
you
know
experienced
with
something
that
is
not
yet
reliable.
So
we
released
it.
We
we
have
some
friends
from
the
my
school
community
who
helped
us
review
this
prior
to
release.
But
it's
it's
not
completely
in
the
open.
It's
Ghaleb
dot-com
slash,
get
up,
slash
ghost
it's
in
there
somewhere.
A
We
really
encourage
your
collaboration.
We
have
there's
some
very
nice
users
to
ghost
who
are
willing
to
help
in
testing
it
on
Amazon,
RDS
or
in
multi-source
purification,
and
we're
giving
good
valuable
feedback
and
we're
so
grateful
to
them,
and
we
are
grateful
for
any
contribution
if
you
can
help
with
documentation.
That
would
be
awesome
because
you
know
how
documentation
takes
like
sixty
percent
of
your
time,
you're
like
you're
coding,
forty
percent
and
then
there's
more
documentation.
Time
then
coding
time,
and
if
you
want
to
submit
a
pull
request.
A
Please
do
let's
discuss
publicly
in
issues.
Tell
us
what
you
want
to
do,
there's
actually
some
limitations
to
goes
that
can
be
solved,
but
not
on
our
immediate
roadmap,
and
we
offer
these
hey.
If
someone
from
the
community
wants
to
help
out,
we
don't
have
the
time
to
do
that
right
now
and
it's
not
on
our
roadmap,
but
if
it
wants
to
compute
that,
please
do
so.
Please
do
give
me
a
shout
out
any
one
of
us
open.
An
issue
suggest
we're
happy
for
contributions.
A
Basically,
the
limitations,
there's
a
list
of
limitations
I
just
wanted
this
light
to
be
there,
so
that
no
one
says:
oh
hey.
Everything
is
great,
so
there's
no
support
for
foreign
keys,
but
it's
open
for
contributions.
This
can
be
done.
It's
not
such
a
big
change.
It
can
be
done.
No
support
for
triggers
can
be
done.
No
support
for
5-7
generator
comb.
It
can
be
done.
We're
moving,
will
we're
thinking
of
our
be
are
minimal
format.
A
A
Wow,
that's
a
long
story,
so
so
I
happen
to
be
the
original
author
of
the
open
article.
My
schema
change
tool,
which
is
the
original
schema
migration
tool
which
was
later
forth
by
for
Facebook
and
I
know
they
I
had
some
influence
in
in
the
in
the
way
they
did
their
own
solution
and
in
fact,
at
the
beginning
they
did
want
to
do
a
big
log
base
solution.
It's
just
that
at
the
time,
bin
logs
were
only
stating
base.
A
Row
base
was
relatively
new,
which
meant
the
tool
would
have
worked
by
parsing,
SQL
queries
and
hoping
to
make
that
correctly,
and
that
would
have
posed
several
limitations
on
the
tool.
So
they
built
something
relatively
similar
to
to
a
trigger
based
solution,
and
they
just
kept
with
it
for
years
because
it
works
for
them
and
I
know
there
right
now,
rewriting
something
in
Python
and
I.
Don't
know
the
exact
details,
but
it's
basically
a
simple
rewrite
and
no
way
no
ways
is
it
as
elaborate
is.
A
A
Do
we
see
realize
all
the
rights
to
all
the
tables?
That's
what
the
binary
logs
do.
Naturally,
the
binary
laws
are
streamlined.
Events
of
all
rights
to
all
tables,
so
I'm
past
time
and
I
want
to
be
polite
to
the
next
speaker,
so
I'm
gonna
be
around
here
at
the
back.
If
anyone
wants
to
ask
for
the
question,
thank
you
very
much.