►
Description
Join us for part one of a three part tech talk series: Upgrading from legacy to the cloud with Scribd. This series will run August 24-Aug 26 at 9AM PT each day. Come join us!
Part One: Replicating Data to the Cloud
Alexander Kushnir (https://www.linkedin.com/in/alexander-kushnir-2b96114a/) and Maksym Dovhal from Scribd discuss with Denny Lee from Databricks the technical and business issues when migrating Scribd’s systems from legacy environments to the cloud. We discuss many technical issues ranging from S3 eventual consistency, cross-cloud consistency, metastore consolidation issues, utilizing multiple catalogs for the same Delta tables, data replication from on-premises to the cloud, and more. Listen to how Scribd engineers used Delta Lake to solve their production distributed cloud data issues. Databricks is proud to announce that Gartner has named us a Leader in both the 2021 Magic Quadrant for Cloud Database Management Systems and the 2021 Magic Quadrant for Data Science and Machine Learning Platforms. Download the reports here. https://databricks.com/databricks-named-leader-by-gartner
A
Is
danny
lee
the
great
engineering
folks
from
script,
to
talk
about
specifically
replicating
replicating
data
into
the
cloud?
To
give
you
a
little
context,
my
name
is
denny
lee,
I'm
a
staff
developer
advocate
here
at
databricks,
I've
been
working
with
apache
sparks
since
about
0.6
and
delta
lake,
since
probably
0.3
or
0.2,
and
prior
to
that
I
was
at
microsoft.
A
One
of
the
projects
that
I
was
part
of
was
project
isotope,
which
brought
hadoop
into
windows
and
azure.
So
you
can
partially
blame
that
on
me,
so
my
apologies,
but
nevertheless
the
key
context
here
is
not
so
much
about
myself,
but
it's
actually
about
our
engineering
friends
here
at
script.
So,
let's
start
with
alexander
alexander,
want
to
introduce
yourself
first.
B
A
C
A
Perfect,
perfect,
okay!
Well,
this
is
a
great
start,
great
set
of
introductions.
So
let
me
go
ahead
and
start
in
terms
of
when
we're
talking
about
replicating
data
to
the
cloud.
Let's
get
some
context
first,
okay
about.
Why
and
what
are
the
processes
that
script
had
to
go
through
to
replicate
data,
the
crowd
so
alex?
I
think
I'm
supposed
to
start
with
you,
but
max.
Please
definitely
chime
in
if
you
have
any
thoughts
or
call
or
color
commentary
where
were
was
alex.
B
All
right,
so,
where
are
we
coming
from?
When
we
took
on
the
adventure
of
this
cloud
migration?
We
started
with
a
what
I
would
describe
as
a
standard
hadoop
on-premise
infrastructure
we
had
a
large
cluster
with
which
was
hosting
our
hdfs-based
data
lake
and
the
same
cluster
was
used
for
the
purposes
of
nightly
compute,
ad-hoc
development
and
etc
so
yeah.
B
B
We
had
lots
of
tables
with
small
file
problems
on
them.
We
have
variants
in
our
file
sizes
for
the
same
tables.
We
also
had
variants
in
our
compression
algorithms.
We
even
had
tables
that
used
three
different
compression
algorithms
for
different
partitions
that
they
had
so
yeah.
This
is
where
we're
coming
from
and
we
didn't
want
to
bring
all
this
legacy
along
with
us.
We
wanted
to
use
this
as
a
true
upgrade
opportunity
and
move
into
a
much
cleaner,
much
more
uniform
storage.
A
Wow,
okay,
so
there's
a
lot
to
unpack
here.
Actually,
so
let
me
go
ahead
and
dive
into
some
of
those
issues
just
because
you
know
we
could
probably
even
just
go
on
for
a
couple
hours
just
on
all
these
issues.
But
let's
start
with
the
the
concept
of
when
you
were
working
on-prem
with
your
hadoop
environments
right,
you
noted
the
fact
that
you
wanted
to
actually
have
independent,
compute
and
storage
that
you
wanted
to
decouple
those
two
concepts
from
your
perspective
like.
Why
was
that
so
important
for
you
was
it?
A
B
B
So
if
we
wanted
to
add
hdfs
space,
we
were
paying
for
storage
and
for
additional
cpus
and
memory.
We,
we
didn't
really
have
the
workloads
to
fully
utilize
our
computer
resources
hundred
percent
at
the
end
of
the
life
of
that
system.
So
we
had
much
more
cpu
and
memory
that
we
could
have
used.
Another
problem
was
that
we
have
a
lot
of
jobs
that
run
in
parallel
on
our
infrastructure
and
the
nature
of
those
jobs.
B
B
A
Oh,
no,
that
actually
makes
a
ton
of
sense.
I
actually
in
my
past
one
of
my
past
environments.
We
had
a
we.
I
was
trying
to
convince
our
operations
team
that
we
needed
to
set
up
separate
routers
for
our
hadoop
cluster
and
but
they
didn't
decided
that
they
didn't
want
to
do
it.
So
when
we
were
running
a
very
simple
terror,
some
terror
sort
create,
we
basically
were
doing
a
ddos
attack
into
our
internal
network.
So,
yes,
completely
grok,
where
you're
coming
from.
C
A
Network
isolation
and
the
needing
the
fact
that
you
weren't
utilizing
the
full
compute
clusters,
but
then
I
I
guess,
because
of
all
these
shuffles,
that
you're
running,
like
the
joins
the
groups
or
whatever
group
I's
that
you're
doing
this.
Is
this
the
main
reason
for
the
small
file
problem
or
the
variance
of
file
sizes
or
the
various
variants
and
compression
algorithms
that
you
were
facing
in
your
legacy
system?
Or
is
that
just
due
to
other
factors?
I'm
just
curious
different.
B
Factors
contributed
to
different
problems.
Our
small
file
problems
were
mostly
from
somebody
trying
to
run
a
big
join.
It
doesn't
work
developer,
the
first
thing
that
would
or
data
scientists
or
whoever
is
doing
this.
The
first
thing
they
will
try
is
they
will
pump
up
sql
shuffle
partitions
configuration
from
200
to
5000.
B
The
job
will
run
through,
but
what
they
don't
realize
is
that
now
they
are
writing
out
5000
files
at
the
end
of
this
job
without
first
reorganizing
the
data
before
dumping
it
to
the
disk.
We
also
variants
and
file
sizes
came
from
us
loading
data.
From
let's
say
we
were
using
scoop
to
load
my
sequel
data
and
the
way
school
partitions
data.
If
there
is
any
kind
of
data
skew
on
on
the
primary
key
in
the
source
tables,
it
will
exist
in
your
target
table.
B
So
this
is
where
that
was
coming
from
variance
in
in
file
formats.
I
think
it
was
just
we
were
using
so
many
different
tools.
We
were
using
spark
hive
impala
scoop,
each
one
of
them
has
its
own
set
of
defaults
and
nobody
ever
bothered
to
go
override
any
of
these
defaults
to
to
make
a
uniform
storage
structure
right.
So
we
we
had
a
little
bit
of
everything.
A
Oh
again,
I
completely
understand
where
you're
coming
from
about
the
fact
that
the
different
tools
resulted
in
different
things,
different
conversations,
different
default
parameters.
How
bad
was
your
small
file
problem?
I'm
just
curious.
I
know
like,
for
example,
my
past
environments.
Whenever
I
had
like
these
multi-tool
problems
would
end
up
happening.
A
Is
I
I
end
up
having
like
hundreds
or
thousands
of
these
like
64k
files
or
or
smaller
generated,
and
they
wouldn't
run
the
compaction
job
or
anything
like
that,
and
so
we
would
just
have
these
like
millions
of
small
little
files,
of
course
that
just
slowed
down
our
hadoop
queries
massively
when
we
did
that,
I'm
just
curious.
If
you
were
going
through
the
same
thing,
it's.
B
Been
tens
of
millions
close
to
hundreds
of
million
of
files
on
our
cluster
and
our
name
nodes
had
to
be
pretty
beefy
to
deal
with
that,
so
we
were
by
the
time
we
were
moving
away
from
our
legacy
infrastructure.
We
were
getting
pretty
close
to
having
to
deal
with
this
issue
rather
than
wanting
to
deal
with
this
issue,
so.
A
Got
it
got
it?
Okay?
So
yes,
so
you
definitely
were
suffering
from
the
same
set
of
issues,
okay,
so
in
terms
of
the
small
file
problem.
So
but
then
how
does
shifting
to
the
cloud
necessarily
solve
that
problem?
After
all,
if
you're
using
the
same
set
of
tools,
you
would
still
technically
have
the
same
problem
so
like
so
it
wasn't
just
about
switching
to
the
cloud
or
I
presume,
as
you
were
coming
from
the
legacy
systems
that
you
actually
decided
that
you
needed
to
go
ahead
and
started
migrating
like
the
processes
as
well.
A
B
One
thing
that
we
were
lucky
with
here
is
that
we
didn't
really
have
much
direct
hdfs
access
anywhere
in
our
workloads.
All
of
our
data
was
accessed
through
a
hive
table
interface.
So
all
the
queries
ended
up
just
going
and
first
resolving
the
hdfs
location
in
age,
catalog
and
then
performing
the
underlying
analytics.
B
So
what
it
allowed
us
to
do
is
as
long
as
we
keep
those
sql
interfaces
sql,
I
guess
whatever
catalog
interfaces
in
place,
we
can
go
and
change,
underline
storage
format,
file
layouts,
whatever
we
want
to
do
and
still
support
our
workloads.
So
it
was
a
bit
of
a
blessing
for
us
that
we
had
that.
So
it
helped
us
a
lot
to
to
move
in
a
desired
direction.
A
Right,
so
actually
that's
that's
something.
I
definitely
would
love
to
dive
a
little
bit
on.
So
in
terms
of
those
options,
you
went
through
various
different
options
and
decided
that
one
of
the
underlying
factors
was
this
sql
slash,
match
catalog
perspective.
That
was
super
important.
Is
it
because
of
all
your
customers
were
like
your
typically
your
bi
tools,
or
is
it
because
they're
typically
data
scientists,
or
is
it
just
because
the
fact
that
you're
intermixing
all
of
these
different
things
and
that's
the
reason
why
that
ultimately
became
sort
of
a
standard
for
you.
B
B
This
is
what
we
inherited
when
we
took
over
the
system
is
this
was
one
thing
that
actually
was
great
and
made
this
transition
so
much
easier
for
us,
so
there
were
some,
of
course
there
were
some
edge
cases
where
people
would
go
and
work
with
hdfs
directly
and
and
write
per
k
directly
into
hdfs,
but
those
were
so
far
few
in
between.
We
could
just
manually
transition
them
later.
A
B
So
yeah,
so
so
so
we
we
we
looked
at
that
and
we
because
of
that
we
started
actually
relating
multiple
options.
So
one
of
the
options
that
we
considered
was
well:
let's
keep
our
catalog
as
is,
and
just
copy
the
data
to
cloud
and
then
point
our
catalogs
to
s3,
and
so
basically
the
table
customers
that
used
to
be
on
hdfs
will
still
be
table.
Customers,
it's
just
going
to
point
to
s3.
Now
and
and
theoretically,
everything
should
work,
but
what
we
quickly
realized
is.
There
is
like
two
fundamental
problems
with
that.
B
The
first
fundamental
problem
is
cost
it's.
We
would
actually
be
doing
so
much
data
transfer
between
data
centers
because
the
data
is
in
s3,
but
the
actual
workloads
running
our
in
our
data
centers
that
it
will
become
cost
prohibitive
and
another
problem
that
we
quickly
realized
is
going
to
bog
us
down
would
be
eventual
consistency
of
s3
and
the
fact
that
we
weren't
just
moving
the
data
for
it
to
sit
there.
B
We
were
later
on
planning
on
moving
some
compute
workloads
would
mean
that
some
data
would
be
computed
in
one
data
center
using
one
hadoop
vendor.
Well,
other
data
sets
would
be
computed
using
data
bricks
in
a
different
cloud
and
trying
to
establish
this
cross-vendor
cross-cloud
eventual
consistency.
S3
storage
is
actually
quite
an
undertaking,
so
we
skipped
that
and
when
we
were
looking
at
different
ways
to
solve
eventual
consistency
with
s3,
we
quickly
realized
how
how
many
problems
delta
solves
for
us.
B
A
Got
it?
No,
that
makes
a
ton
of
sense,
and
so
before
I
switch
over
to
max
to
dive
a
little
bit
deeper
about
the
solution.
I
did
want
to
talk
a
little
about
the
nuances,
about
eventual
consistency
in
s3.
So
for
anybody
that's
heard
me
talk
about.
You
know
delta
lake.
They
would
already
know
that
I
would
just
repeat
the
mantra
of
asset
transactions.
A
You
know
the
atomic
transactionality,
but
I
wanted
you
to
sort
of
bring
it
real
like
what
were
the
issues
when
it
came
to
eventual
consistency
for
you
that
was
actually
gonna
like
what
were
the
problems
that
eventual
consistency
would
cause
right
outside
of
just
cost.
Obviously
there
was
the
cost
issues,
but
I'm
just
curious
like
was
it
because
people
were
ultimately
querying
dirty
snapshots,
I'm
just
curious
from
your
perspective.
What
were
those
factors
that
might
have
been
coming
into
play
here?
A
B
B
This
problem
is
becomes
even
more
of
a
problem
when,
when
you
overwrite
data-
and
we
have
lots
of
jobs
that
would
instead
of
just
adding
a
new
partition,
they
would
overwrite
multiple
partitions
worth
of
data
and
that
basically
makes
that
problem.
So
much
worse,
so
we
we're
looking
at
various
options
like
we
could
have
used.
We
looked
at
s3
guard
right
as
one
of
the
options,
but
the
problem
with
this
regard
is:
it
doesn't
really
work
when
you
have
multiple
clusters,
like
it
works
for
all
the
writes
happening
from
the
same
cluster.
B
But
if
you
try
to
share
this,
let's
regard
dynamodb
between
multiple
clusters,
things
start
breaking
and
it
wasn't
a
use
case
that
was
probably
very
well
tested
and
addressed.
So
we
shied
away
from
it
plus
the
solution
we
ended
up
with
was
so
much
better
and
allows
us
to
just
move
faster
in
this
further
phases
of
our
upgrade.
A
Oh,
that's
really
cool
to
hear
so
yeah,
so
the
the
the
context
of
of
basically
this
data
reliability
really
allowed
you
to
focus
on
the
real
problems,
because
you
were
sure
of
the
transactional
requirements,
the
protection
of
your
data,
I'm
just
curious.
This
is
just
more
of
a
fun
question.
Did
you
actually
have
to
worry
about
like
deletion
of
entire
partitions
or
tables,
but
then
at
least
you
were
able
to
roll
back
to
it
like
the
old
oops
scenarios,
or
were
you
able
to
avoid
most
of
those
I'm
just
curious.
B
A
B
A
C
A
Excellent,
no,
no!
It's
a
great
call-out!
No!
No,
because
in
a
lot
of
ways,
that's
that's
pretty
much
the
reason
why
we
had
created
delta
in
the
first
place,
because
we
end
up
having
customers
going
like
dude.
How
do
you?
How
do
I
roll
back
or
undelete
what
I
just
did?
Dang
it?
It's
like
you
just
deleted,
like
three
petabytes
of
data
yeah,
we're
gonna
have
to
find
an
easier
solution
to
restore
from
tape
or
glacier
or
whatever
it
was
so
yeah.
No,
no
worries,
okay!
A
So
in
that
segway
max,
like
let's
talk
a
little
bit
more
about
like
you
know
what
you
ended
up
on
you,
we,
we
can
start
talking
about
now,
you're
in
the
cloud
now
you're
using
spark
now
you're
using
delta.
Why
don't
you
talk
a
little
bit
about
like
how
you
simplify
the
tooling,
how
you
made
that
transition
things
of
that
nature.
C
Yes,
so
we
we
had
few
problems.
Yes,
so,
first
of
all,
we
need
to
to
to
have
two
that
will
be
fully
automated,
because
we
don't
want
to
stop
business.
We
don't
want
to
stop
new
table
creation
or
something
like
this,
so
we
need
fully
automated
tools
that
will
search
our
legacy.
Catalog
define
find
the
data
new
newly
created
data,
all
data
modified
data
and
copies
this
data
to
new
infrastructure,
but
also
we
don't
want
to
copy
the
css.
C
C
C
A
Yeah
absolutely
yeah,
so
the
absolutely
so
I'd
love
to
understand
a
bit
better
about.
How
did
you
keep
those
catalogs
in
sync,
because
that
is
usually
a
problem
for
most
people
right,
especially
because
I
think
one
key
point
that
we've
sort
of
implied
but
haven't
actually
stated
very
clearly,
is
that
your
migration
to
the
cloud
it
wasn't
just
like
done
in
one
full
step?
A
It's
not
like
you
just
like
you
know
one
day,
you're
on
on-prem
the
next
day,
you're
in
the
cloud
and
you've
turned
off
everything
on-prem
right
like
I'm,
I
guess
I'm
going
to
start
with
that
problem.
How
long
was
that
transition
from
the
cloud?
Sorry
from
on-prem
to
cloud?
How
long
did
it
actually
take,
because
this
obviously
wasn't
done
in
one
day.
C
Yes,
it
definitely
was
done
in
months,
not
months
how
it
was
we.
There
was
two
problems:
first
problem:
it
was
initial
copy
of
this
like
petabyte
of
data
or
something
like
this,
and
the
next
problem
was
daily
incremental
copy
that
we
need,
because
we
have,
we
migrate,
our
computation
one
step
by
step.
So
there
was
some
time
when
we
have,
for
example,
half
computation
on
old
cluster
and
sync
with
in
the
middle
and
other
computation
on
new
infrastructure,
so
how
it
was
realized.
C
C
Second
part
was
so
when
we
have
the
list
of
tables
that
we
need
to
copy.
We
started
our
data
sync
application
and
it
lists
all
tables
in
hadoop
and
a
list
of
partitions
and
and
files
that
exist
on
hdfs.
So
it
was
like
we
use
like
checksum
for
updated
data.
It
was
combination
of
file,
name,
modification
time
and
file
size.
C
So,
if
all
so
how
we
decide
what
we
should
copy,
if
this
combination
of
file,
size
file,
name
and
modification
time
was
changed,
we
should
copy
this
data
to
aws
to
track
this
information
on
aws
site.
We
have
a
additional
like
metadata
table
where
we
wrote
all
copied
data,
so
we
for
all
pipeline
looks
like
we
list.
For
example,
one
table
collect
all
files.
C
A
Got
it
so
then
to
make
sure
I'm
on
the
same
page?
Basically,
you
have
this
data
sync
code
that
you
created
that
allows
you
to
look
within
hdfs
to
figure
out
what
files
changed
for
the
tables
that
are
listed
right
and
then
copy
that
data
up
to
s3.
Accordingly,
when
you
copy,
oh
sorry,
go
ahead
and
convert
to
delta,
so
there's
actually
a
couple
of
parts
that
I'd
love
to
understand,
better
rob.
So
for
starters,
when
you
got
that
list
like
that,
typically
was
just
more
like
that
was
an
easier
read.
A
I
take
it
like
it's
just
more
like
query
the
two
catalogs
between
what
was
on-prem
and
what
was
in
the
cloud
and
make
sure
that
they
match
and
they
and
whatever
matches
a
good
old-fashioned
join.
Those
are
the
tables
that
you'd
look
at.
Then
you
listed
out
the
files,
so
I'm
just
curious
like,
for
example,
you
know
how,
with
cloud
object,
stores
listing
files
can
be
really
really
painful
right.
C
No
no
with
this,
which
interface
we
can
list
it
quite
fast,
because
name
node
of
gfs
consists
all
this
information.
So
it's
really
fast.
Of
course,
we
used
some
multithreading
to
to
do
it
faster
because
in
one
thread
it
will
be
super
slow,
but
it
was
quite
fast.
We
can
can
do
it.
It's
not
s3
for
hdfs,
okay,.
A
Oh
no,
this
is
great.
I
mean
it's
a
great
example
of
example:
the
the
difference
between
the
how
the
file
structure
worked
when
you're
working
with
hdfs
versus
the
cloud
and
actually
how
you
have
to
think
differently
because
of
it.
So
that's
actually
the
only
reason
I'm
calling
this
out.
Okay,
so
perfect,
so
at
least
getting
the
files
figured
out
was
really
easy
and
fast
on
the
hdfs
side.
So
perfect.
Now
you
have
the
differences
for
each
table.
A
C
C
It
looks
like
very
simple,
but
if
you
want
to
do
it
in
really
high
speed
with
a
lot
of
data,
it
should
be
really
optimized
that
you
should
think
about
the
the
number
of
output
files,
compressed
small
files
or
utilize
resources
for
100
percent,
like
your
application,
should
not
wait.
Just
copy
copy
copy
so.
A
Right
right,
so
so
just
to
confirm
make
sure
everybody's
on
the
same
page.
You're
then
copying
your
data
sync
tool
is
basically
copying
the
files,
the
the
the
change
files
or
the
new
files.
Excuse
me
from
your
on-prem
environment
for
those
tables
copies
them
over
to
the
cloud.
At
that
point,
then,
basically,
it's
just
a
spark
job
that
reads
it
and
then
because
you've
optimized
it
and
you've
compacted.
So
that
way,
you're
tr
you're,
avoiding
the
small
file
problem
and
then,
when
you
ins,
you
basically
insert
that
data
into
your
delta
table.
A
C
We,
when,
when
data
was
updated
on
old
premier
on-premise
cluster,
we
overwrite
full
partition
of
full
table,
so
we
don't
use
merge
because
it's
slow,
first
of
all
and
second
merge
required
actual
data
read
from
aws
and
we
should
pay
for
output
traffic.
So
we
just
have
like
metadata.
If
metadata
is
not
the
same,
that
we
have
in
on-premise,
we
just
overwrite
full
partition
and
it's
okay.
A
C
Copy
was
quite
fast,
but
why
it
was
something
like
one
month
because
to
make
this
copy,
we
should
utilize
our
on
purpose
cluster
and
we
should
not
block
jobs
that
related
to
business.
So
we
we
start
actually
actual
copy
was
started
only
when
all
daily
pipeline
was
finished.
So
when,
when
cluster
was
free,
that's
why
it
was
not
like
one
one
huge
copy
for
few
days.
It
was
something
like
five
six
hours
per
day.
C
A
B
I
think
there's
any
kind
of
transitioning
there
will.
There
will
be,
I
guess,
educational
gaps.
People
need
to
understand
them,
but
they
definitely
don't
ask
us
anything
about
underlying
storage
of
files
or
anything
because
they're,
just
working
with
tables
right,
it
was
transparent
to
them,
and
that
was
the
goal
for
us
to
to
get
to.
That
point
is
yes,
we
did
have
couple
jobs
that
were
writing
to
hdfs
directly.
I
told
you
about
those
and
those
guys
had
to
be
educated.
B
A
Oh,
that,
that's
actually
really
awesome
to
hear.
So.
Yes,
we'll
definitely
go
ahead
and
be
promoting
that
when
it
when
it
comes
out
just
because
I
feel
that
they're
going
to
be
plenty,
customers
that
have
p
p,
p
practitioners
and
customers
that
actually
have
followed
very
similar
footsteps
as
yourself
right,
which
is
they
have
to
take
their
on-prem
environments.
They
need
to
move
it
to
the
cloud
and
there's
a
bunch
of
hiccups
on
the
way.
A
So
then
I
I
guess
you
know
in
the
final
bit
of
this
particular
session-
are
there
any
little
tidbits
or
best
practices
or
tips
and
tricks
that
you'd
like
to
call
out
when
you
did
that
transition
like,
for
example,
like
like
one
exam?
What
I
mean
by
that,
for
example,
is
like
that
little
our
discussion
about
the
fact
that,
like
file
listings
in
hdfs,
actually
is
fast
in
all
sources
and
then
switching
to
the
cloud.
A
Not
so
much
right.
In
fact,
that's
why
delta
places
the
the
list
of
files
directly
in
the
transaction
log?
So
you
can
just
query
that
and
that's
significantly
faster
than
asking
you
know,
s3
to
list
out
all
of
its
files.
I'm
just
curious,
were
there
any
other,
you
know
pros
cons,
doesn't
matter,
but
just
tips
and
tricks
or
best
practices
from
when
you
went
ahead
and
did
that
transition
yourselves.
A
B
Mean
one
thing
I
can
think
of
is
if
you're
moving
to
an
object,
store
based
data
lake,
just
just
just
go
with
delta,
just
like
we
we
wanted,
because
I'm
like,
I
might
sound
like
hey,
I'm
trying
to
promote
something.
I'm
not.
It
actually
did
solve
a
lot
of
problems
for
us
and
this
whole
concept
of
a
transaction
log
of
on
neo4j
files.
It
just
unlocks
so
many
features
and
I'm
pretty
sure
some
of
the
features
haven't
yet
been
realized,
and
so
it's
we
chose
it
for
one
set
of
reasons.
B
B
You
heard
another
great
thing
about
it.
Like
hey,
I
screwed
something
up
when
we
were
doing
this
data
sync,
I
actually
messed
something
up
in
the
source
and
that
got
and
data
sync
replicated
that
that's
actually
one
good
piece
of
advice
when
your
data
sync
in
one
way
make
sure
you
keep
your
source
data
clean
right.
If
you
delete
something
on
the
source,
our
data
sync
would
actually
also
delete
it
in
the
target.
B
Luckily,
I
timed
it
very
well
because
just
like
three
or
four
days
prior
to
that
delta
did
come
out
with
the
restore
feature
that
was
that
made
it
much
easier.
If
it
wasn't
there,
it
would
have
still
still
be
possible
because
we
could
still
read
the
data
from
the
past
overwrite
it
and
write
it
back
into
the
same
delta
table.
It's
just
made
it
much
clearer,
so
yeah.
A
A
C
Just
want
to
add
that
if
you
decided
to
to
move
operate,
your
system,
don't
worry
and
do
it
with
this
new
technologies.
Don't
move
your
old
stuff
to
the
cloud.
There
are
a
lot
of
new
interesting
technologies
frameworks
formats
and
if
you
decided
to
switch
to
cloud,
it
doesn't
mean
that
just
move
your
old
old
infrastructural
data
format
and
all
your
problems
to
a
clown
leave
it
in
in
all
on-premise
cluster
and
move
with
a
fresh
new
technologies
to
the
new
infrastructure.
A
Max,
that's
actually
really
great
advice.
I
I've
seen
this
happen
time
and
time
again
where
people
it's
like,
we'll
just
do
the
exact
same,
like
templates,
that
we
did
for
on-prem
and
we'll
just
apply
it
to
the
cloud
and
magically
everything
gets
fixed
and
you're
going
not
really
you're
missing
that
entire
elasticity
independence
thing
that
you
actually
need
to
have
right,
so
com,
great
advice.
B
One
of
the
patterns
that
we're
actually
trying
to
push
on
people
is
changing
how
they
develop,
because
previously
the
way
they
would
test
their
changes
is
they
would
copy
a
table
into
a
different
name.
Space
like
they'll
copy,
the
table
into
their
personal
database.
They'll
change
the
job
to,
instead
of
writing
to
original
table,
to
write
to
this
other
table
and
now
they're,
invoking
it
checking
the
outputs
etc.
That's
that's,
basically
the
process.
B
What
we're
unlocking
now
is
telling
people
hey.
Instead
of
having
this
one
centralized
catalog,
we
can
have
multiple
catalogs
and
our
developers
can
without
changing
the
code
of
the
job,
just
change
the
catalog
definition
and
point
the
table
to
an
alternate
location
run
their
job,
as
is
without
any
code
changes
you
can.
You
now
have
data
in
two
places.
You
can
still
compare
it.
You
like
everything,
well,
you'll,
just
deploy
it
and
then
production
catalog.
It
will
point
where
it
needs
to
point
so
this
pattern.
B
A
A
Well,
this
is
my
test
environment,
one
in
u.s,
east,
one
versus
you
know,
in
my
case
usd
u.s
west,
2
or
whatever
else
that
I
was
using
and
then
and
then,
if
I
forgot
to
do
it
in
one
place
and
then
also
I'm
like
accessing
one
one
like
application
in
the
wrong
region
in
the
wrong
in
the
wrong
testing
environment
or
whatever
else
so.
Yes,
the
it
seems
to
simplify
things.
A
lot
just
right
now,.
B
Yeah,
simplify
is
the
keyword
here
because
fundamentally
you're
you're,
adding
all
of
this
additional
interfaces,
and
you
have
to
code
around
all
of
these
additional
interfaces
in
your
app.
But
at
the
end
of
the
day,
you
still
whether
you
change
configuration
or
you
change
code,
you're
still
pushing
something
different
into
a
different
environment.
A
C
I
already
told
a
lot
what
what
I
want
to
say.
Yes,
what
I
want
to
say
is
that,
with
the
new
technologies,
the
development
process,
it
is
much
easier
because
we
have
a
lot
of
built-in
functionalities
that
help
us
to
store
data
properly.
Read
data
fast,
join
data
faster
optimize
on
runtime
on
something
like
this.
A
Perfect,
actually,
this
is
an
excellent
way
to
finish
off
today's
session
again
max
alex.
I
really
appreciate
you
guys,
taking
time
out
of
your
busy
schedule
to
join
for
this
particular
session
on
migrating
to
the
cloud
or
upgrading
to
the
the
cloud
so
again,
thank
you
very
much
for
taking
your
time.