►
Description
Join us for part two of a three part tech talk series: Upgrading from legacy to the cloud with Scribd. Part Two: Moving Batch Jobs over to the Cloud.
Part One: Replicating Data to the Cloud Recording: https://youtu.be/vGv6AcPp7Zs
Abstract: Alexander Kushnir and Stas Bytsko from Scribd discuss with Denny Lee from Databricks the technical and business issues around moving batch jobs to the cloud as part of Scribd’s migration from legacy environments to the cloud. Instead of performing the migration as a big bang, there was a byte-by-byte migration performed incrementally to minimize any disruption to the business. Listen to how Scribd engineers used Delta Lake to solve their production distributed cloud data issues.
Speakers:
Stas Bytsko is the team lead of the Data Engineering team at Scribd with somewhat mysterious, and possibly exciting, past.
Alex Kushnir is a Data Architect and a Tech Lead of Data Engineering Team @ Scribd. Throughout his almost 20 year career, he has acquired experiences in various Software Engineering domains: desktop applications, web development, mobile APIs, distributed computing, cloud architecture, big data. He designed and implemented solutions utilizing various data stores: relational databases, document databases, key/value stores, object stores. For the past 5 years, he focused on distributed computing in cloud environments utilizing various BigData tech stacks and he’s a big fan of Apache Spark.
https://www.linkedin.com/in/alexander-kushnir-2b96114a/
Denny Lee is a developer advocate at Databricks, where he works on Delta Lake, Apache Spark, Data Sciences, and Healthcare Life Sciences. He has previously built enterprise DW/BI and big data systems at Microsoft including Azure Cosmos DB, Project Isotope (HDInsight), and SQL Server as well as the Senior Director of Data Sciences Engineering at SAP Concur. Denny holds a Masters in Biomedical Informatics from Oregon Health Sciences University. Databricks is proud to announce that Gartner has named us a Leader in both the 2021 Magic Quadrant for Cloud Database Management Systems and the 2021 Magic Quadrant for Data Science and Machine Learning Platforms. Download the reports here. https://databricks.com/databricks-named-leader-by-gartner
A
B
Hey
guys,
I'm
alex
in
case
you
haven't
heard
of
me
yet
which
you
should
have,
because
you
should
have
watched
the
previous
video,
I'm
a
software
architect,
a
data
architect
at
script
and
I'm
a
tech
lead
on
data
engineering
team.
I
have
a
variety
of
experience
in
my
years
prior
to
that,
but
for
the
past
somewhere
around
five
years,
my
focus
has
been
big
data
and,
as
I
mentioned
before,
a
big
fan
of
spark.
A
C
Hello,
my
name
is
tas.
I'm
a
team
lead
on
the
data
engineering
team.
I
work
with
alex.
He
provides
the
architecture
for
the
migration
and
we
with
the
team,
implement
it.
So
I've
worked
with
for
several
years.
I
worked
as
a
data
engineer
and
before
that
I
was
a
backend
engineer
and
data
engineering
is
much
more
fun.
I
can
tell
you
that.
A
Excellent
excellent
thanks
very
much
toss,
so,
okay!
Well,
if
you
don't
already
know
who
I
am
my
name-
is
danny
lee,
I'm
actually
with
databricks,
so
I'm
different
from
these
guys
with
the
nice
black
hoodie.
My
past
is
in
addition
to
the
fact
that
I'm
a
developer
advocate
here
at
databricks
in
the
past.
I
was
actually
on
the
team
at
microsoft
that
created
what
now
is
known
as
azure
hdinsight.
A
The
reason
I
bring
that
little
topic
up
is
because
we
are,
as
this
is
part
of
the
series
of
upgrading
from
legacy
to
the
cloud.
So
if
you,
hopefully,
you
actually
watched
the
previous
video
session.
Excuse
me
with
also
alex,
as
he
alluded
to,
in
where
the
script
was
describing,
how
they
upgraded
from
their
legacy,
hadoop
and
hive
environments
to
to
the
cloud
also
using
apache
spark
and
various
other
tools.
A
B
B
We
had
to
move
our
hdfs
data
to
s3,
and
on
top
of
that,
we
actually
had
to
keep
the
data
in
sync
afterwards,
because
we
didn't
do
a
one-day
migration.
It
actually
took
us
a
little
bit
under
a
year,
but
if
you
take
into
consideration
all
the
planning
and
prep
work,
it
actually
took
us
a
little
bit
over
a
year
to
get
that
done.
B
So
the
process
of
our
data
sync
was
was
pretty
straightforward.
We
were
blessed
with
the
fact
that
there
was
no
almost
no
direct
hdfs
access.
All
the
access
was
done
through
a
catalog
right
through
through
table
interfaces,
and
our
data
sync
process
in
a
nutshell,
was
just
reading
data
from
our
legacy
system
through
table
interfaces
and
writing
them
to
s3
using
table
interfaces.
The
tables
in
s3
were
now
configured
to
use
delta
under
the
hood
so
yeah,
so
that
was
basically
the
process.
B
One
interesting
thing
to
add
to
details
that
we
haven't
talked
about
in
our
last
session
and
it
actually
is
important
in
the
context
of
today's
conversation.
We
don't
shy
away
from
writing
metadata
to
delta
log.
I
mean
write
a
budget
metadata
to
delta
log,
but
one
piece
of
information
that
goes
in
there
is
the
user
of
of
the
transaction.
B
So
the
way
our
data
sync
operator
operations
work
is
that
when
we
go
and
we
figure
out
a
list
of
tables
which
need
to
be
synchronized,
we
also
look
at
the
last
user
who
wrote
to
destination
delta
tables,
and
if
that
last
user
was
not
data,
sync,
our
data
sync
will
automatically
pause
and
take
those
tables
out
of
equation,
because
it
knows
now
somebody
else
touched
them.
So
I
shouldn't
be
synchronizing
this
data,
so
this
will
become
more
clear.
B
A
Wow
that
that's
super
interesting,
okay,
so
then,
let's
dive
into
some
of
those
aspects
right
so
stars,
I
think
you
know
you
had
mentioned
to
me
in
the
pre-conversation
about
things
like,
for
example,
how
you
had
to
you
didn't:
do
the
migration
as
a
big
bank
right
and
as
alex,
had
called
out
right.
There
were
no
major
interruptions.
A
C
So
we
first
thing
we
built
we
built
the
tool
called
validator
which
basically
takes
the
div
of
the
previous,
like
snapshot,
states
of
the
table
and
looked
what
changed
and
compared
the
change
and,
for
example,
like
some
tables,
are
date
partitioned
and
we
would
compare
only
the
latest
date
and
we
did
bunch
of
optimizations.
So
we
like
did
caching
to
reduce
the
amount
of
data
we
had
to
shuffle
around
and
also
reduce
our
data
center
transaction
fees,
and
so
we
use
yeah.
C
We
plan
to
open
source
it,
probably
because
it's
not
that
trivial
from
the
first
look
of
it,
and
so
but
okay,
so
you
have
like
one
legacy
system
task
writing
to
one
output
table
and
you
have
this.
Basically,
you
cannot
and
you
have
the
same
task,
also
different
version
of
it.
I've
written
in
spark,
for
example,
right
into
the
same
table.
So
how
do
we
compare?
We
come
up
with
this
whole
framework
of
called
migration.
C
We
dynamic
we
like
written
like
never
written,
but
we
added
custom,
airflow
operators
which
would
detect.
So
this
is
immigration
table
dynamically
change,
the
output
of
the
tables
and
then
at
the
end
of
their
pipeline,
which
is
huge
beast
more
than
1
000
tasks.
We
would
aggregate
what
tables
were
written
to
and
then
we
just
batch
compared
them,
and
this
is
like
the
one
piece
and
we
discovered
so
many
fun
bugs
and
inconsistencies
just
lists
and
lists
of
them.
Maybe
we'll
share
that
as
well.
A
A
B
This
is
actually
talking
a
lot.
We
had
that,
so
there
we
were
validating
making
sure
that
the
data
that
we
copied
actually
matches
one
to
one,
but
the
next
phase
was
once
we
convert
a
task,
that's
running
in
our
legacy
system
and
it's
now
running
in
our
new
system.
We
want
to
make
sure
that
both
of
them
produce
the
same
output
so
for
a
while
they
cannot
be
writing
to
the
same
output
location.
So
we
created
what
stars
alluded
to
here.
We
created
this
concept
of
a
task
being
in
a
migration
state.
B
B
So
so
now
we
have
one
set
of
data
being
data
synced
from
the
legacy
system,
another
set
of
data
being
computed
on
the
new
system,
and
we
can
compare
them
basically
partition
to
partition
and
we
we
get
to
decide
how
long
of
a
of
a
consecutive
successful
validation
dates
they
have
to
have
before
we're
comfortable
switching
them
from
this
migration
state
to
like,
okay,
now
you're
a
real
task,
you're,
really,
writing
the
outputs
and
the
old
task
is
no
longer
needed
all
right.
So
that's!
A
C
I
just
wanted
to
add
that
our
data
sync,
we
also
well-
it
did
validation
after
that,
but
it
was
so
rock
solid.
We
didn't
find
any
discrepancies,
so
it's
not
that
useful.
If
you
have
yeah,
if
your
code
is
good
enough,
but
when
you
switch
in
frameworks
and
you'll
definitely
find
something.
So
that's
more
of
a
need
for
that.
A
No,
that
makes
a
ton
of
sense,
and
so
actually,
let's
just
unpack
that
little
part
a
bit,
because
this
reminds
me
of
the
old,
like
data
warehousing
days
where,
basically
in
when
we
migrated
from
one
data
warehouse
one
to
data
warehouse
two,
we
actually
basically
ran
two
processes
at
the
same
time
and
we
just
kept
on
overloading
and
then
we
basically
run
a
whole
bunch
of
validation
tests
to
say
hey:
does
the
data
match
between
the
s
the
new
one
versus
the
old
one?
And
so
I
take
it?
A
A
C
Maybe
I
can
respond
so,
basically,
all
the
all
tasks
that
produce
some
output
in
the
data
warehouse
had
to
be
validated
so
as
soon
as
validation
matched.
We
just
dropped
the
task
in
legacy
system
and
just
made
the
task
in
aws
non-migration
that
concluded
the
immigration
for
that
particular
task.
So
it
was
as
fast
as
possible
to
answer
your
question,
but
it
dragged
on
for
quite
a
long
time
because
we
had
to
migrate
a
bunch
of
tasks.
B
I
would
augment
that
with
not
immediately,
but
as
soon
as
we
feel
comfortable,
we
had
some
like
heuristics
for
this,
and
those
heuristics
were
really
if
it's
a
daily
task.
Let's
wait
to
make
sure
it.
It
produces
consistent,
outputs
for
at
least
a
week
right,
so
we're
running
them
in
parallel
and
both
system
for
a
week,
and
we
see
okay,
they
are
producing
output
that
matches
good
to
go.
We
can
switch
them.
B
If
this
is
a
task
that
runs
every
week,
we
we
maybe
will
go
ahead
and
say
well,
203
is
enough,
because
we
can't
really
wait
that
long
and
if
it's
something
like
a
monthly
task,
then
we
we
take
our
chances
a
bit
there
right.
We
will.
Let
have
one
at
most
two
consecutive
match,
matching
validation
runs
and
we'll
move
on
from
there.
A
Right
but
then,
by
the
same
token,
because
you
were
running
so
you
had
many
tasks
that
you
actually
had
to
basically
migrate,
but
then,
from
your
perspective,
you
know
you
were
able
to
do
many
of
them
in
parallel
right.
So
at
least
it's
not
like
you
were
literally
doing
the
searly
one
after
the
other
number.
One.
B
B
B
The
the
the
key
to
this
is.
We
wanted
to
avoid
synchronizing
data
back
from
the
cloud
into
our
legacy
system,
so
the
synchronization
was
only
happening.
One
way,
so
that
meant
that,
once
all
the
data
is
in
the
cloud
and
it's
been
synchronized,
we
can
take
all
of
our
ad
hoc
workloads
and
just
push
them
out
there.
So
we
can
reduce
the
load
on
this
cluster,
maybe
resize
it
a
little
bit
and
but
more
importantly,
we
can.
B
So
when
we're
talking
about
this
workloads
for
our
batch
processing,
as
I
mentioned,
we
have
somewhere
around
1500
jobs.
Interesting
thing
is
that
when
we
started
taking
inventory
of
them
and
saying
who's
going
to
be
migrating,
what
we
realized
that
we
don't
need
to
migrate
like
200
of
them,
I
was
like
man.
This
is
this
is
awesome.
We
should
do
this
more
often
we'll
we'll
clean
up
our
pipelines
significantly
doing
it
that
way.
But
if
you
think
about
the
what
was
what
were
those
workloads,
they
were
basically
initially
you
bring
the
data
into
warehouse.
B
That
would
be
things
like
scoops
that
will
just
bring
your
relational
data
in
we
bring
our
data
from
our
kafkas,
we'll
bring
in
data
from
our
proprietary,
some
other
proprietor,
analytic
systems
that
we're
using.
Then
we
have
a
long
processing
phase,
which
is
basically
the
core,
the
business
logic,
whatever
the
the
meat
of
these
pipelines
and
that's
a
mix
of
clive
jobs
spark
jobs.
There
were
some
jobs
written
in
ruby.
B
That
would
just
like
go
read
the
data
from
hdfs
do
something
with
it
record
at
a
time
and
then
write
it
somewhere
and
then
at
the
end
of
the
pipeline
would
be
what
I
would
refer
to
as
our
output
phase.
At
the
end
of
the
day,
all
of
these
computations
that
happened.
They
have
to
go
somewhere,
so
they
would
either
go
to
tables
that
power
some
kind
of
business
dashboards
or
they
will.
B
We
will
write
this
data
back
out
to
relational
databases
for
the
application
to
use,
or
we
will
go
and
refill
some
kind
of
like
elastic,
search
indexes
or
redis
caches
whatever.
It
is
right,
so
we
we
did
a
bunch
of
computations.
We
inferred
a
lot
of
things
and
we're
now
outputting
it,
so
that
was
the
gist
of
this
big
pipeline.
So
when
we
were
thinking
about,
how
can
we
migrate
it
going
back
to
original
question?
B
A
No,
no
problem
that,
but
that's
actually
a
good
introduction.
We
probably
should
have
actually
done
that
in
the
beginning,
but
yeah.
It
is
what
it
is.
That's
what
happens
when
you
do
a
session
like
this,
so
nevertheless,
this
then
there's
two
other
points
that
I
definitely
want
to
call
out
that
stats
that
originally
called
out
originally
so
in
the
process
of
doing
all
of
the
migration.
All
of
these
tasks
right.
A
First
of
all,
the
fact
that,
even
with
like
maybe
like,
like
you
had
mentioned
before
that
you
know,
maybe
you'll
wait
a
few
days
for
daily
jobs
or
a
few
weeks
for
weekly
jobs
and
so
forth
right.
The
advantage
of
the
approach
that
you
had
is
that,
because
there
are
15,
1300
jobs
that
are
that
you
had
to
move
over
you.
A
lot
of
this
could
be
done
in
parallel.
A
So
that's
where
I'm
going
to
roll
us
back
to
which
is
about
that
automation
right,
you
had
specifically,
the
things
I
think
we
wanted
to
call
out
was
the
automatic
validation
via
custom
airflow
operations
and
your
templated
sql
runner,
because
there
were
especially
from
the
standpoint
of
alex
you
had
mentioned
originally
about
it.
It
was
great
that
you
were
able
to
query
tables
for
most
of
the
stuff,
but
you
had
hive
and
spark
tables
right.
A
C
Here
so
from
those
1
200
tasks
that
we
did
have
have
to
migrate
because
they
did
have
some
business
value.
Unfortunately,
there
were
700
of
them,
which
were
just
plain
sql
templates,
which
were
run
by
five
minus
e
command.
Oh
well,
basically,
first
inside
the
ruby
template
sigma
file,
so
we
had
to
replace
ruby
template
delta
values
with
actual
values.
C
After
that
we
could
run
it
and
the
so
we
decided
okay,
we'll
do
it
where
python
I'm
sorry
like,
we
try
to
run
it
via
airflow,
but
look
like
none
of
the
existing
solutions
did
what
we
needed
to
do
and
we
basically
wrote
two
kinds
of
automation.
First
kind
of
automation
is
because
we
were
running
it
by
hive
and
databricks
platform
does
not
support
hive.
It
only
runs
from
spark,
so
spark
dot,
sql
method
we
couldn't
like,
we
discovered
pretty
pretty
quickly.
C
We
cannot
just
take
that
sql
render
template
and
just
shove
it
into
that
dot.
Sql
and
it
will
go
smoothly.
It
will
not
go
smoothly.
First
of
all,
it
had
to
be
one
statement
at
a
time,
so
we
we
basically
ripped
off
the
parsing
logic
from
five.
Of
course
we
properly
attributed
it
but
yeah,
so
we
parsed
it
yeah
and
after
that
we
just
did
a
bunch
of
manipulations
like
one
of
the
most
obvious
one.
We
used
temporary
tables
everywhere
and
we
couldn't
do
it
because
databricks
only
supports
temporary
view.
C
So
at
the
minimum
we
had
to
replace
those
right,
but
it
wasn't
easy
as
that,
because
temporary
table
three
usually
exists
in
some
database.
The
temporary
view
does
not
exist
in
some
database.
It
you
cannot
put
it.
It's
basically
only
default
database
right,
so
we
had
to
like
dynamically
find
like
is
it?
Is
it
like
temporary
table
in
some
custom
database
or
it
could
be
not
specified?
C
It
could
be
in
the
use
statement
above
so
we
had
to
parse
the
whole
thing
just
find
out
that
all
the
instances
were
reduced
and
just
replace
it,
and
there
could
be
like
multiple
inserts
per
template
and
we
had
to
find
like
all
the
usages
through
the
magic
of
regex,
I'm
actually
actually.
Very.
Very
so.
I
don't
like
that
code.
We're
not
going
to
open
source
it,
because
it's
just
horrible,
but
it
did
the
job
and
with
now
we
don't
need
it
anymore
and
for
everyone
who
just
listening
this
just
use.
C
Hopefully
runnable
spark
template
which
we
could
later
split
again
on
the
semicolon
and
run
with
spark
sql
each
of
those
statements.
One
thing
we
well
that's
very
beneficial.
We
could
have
just
put
everyone
every
of
those
templates
inside
like
spark
class
right
and
just
just
dynamically,
generate
them,
or
something
and
just
hoping
for
the
best.
But
templates
are
very
useful,
especially
for
business
users.
You
say
to
them:
oh,
you
have
to
like
do
it
in
scala
now
and
then,
like
whoa
whoa
whoa
hold
on.
We
didn't
sign
up
for
that.
C
So,
just
to
like
we
wanted
to
be
as
transparent
as
possible.
They
they
shouldn't,
have
felt
the
thing
ever
should
be.
Everything
should
have
been
done
automatically
and
it
is,
and
we
wrote-
and
actually
just
like
parsing
converting
code
and
second
piece
is
the
same
thing.
We
parse
it
again
to
run
it
and
this
time
we
it's
color
compatible
template,
but
a
bunch
of
things
don't
work
exactly
as
we
expected
them
to
be
in
the
scala
data
bricks,
delta
lake,
for
example.
C
C
We
use
the
replace
where
feature
of
the
delta
lake,
when
inserted
into
delta,
to
find
out
exactly
partitions,
yeah,
I'm
not
going
to
details
what
the
dynamic
partition
over
I
just
google,
it
and
and
yeah
it's
yeah,
that's
like
and
we
bunch
of
other
stuff,
for
example
like
well.
I
don't
remember
from
top
of
my
mind
but
there's
like
bunch
of
magic
out
there.
A
No
well,
I
think
everybody
loves,
loves
the
idea.
I
think
you're
actually
calling
out
the
extreme
importance
of
the
fact
that
any
migration
job
ever
created
has
its
own
set
of
idio,
idiosyncrasies
and
issues,
and
so
you're
calling
that
nicely
yes,
regex
is
both
your
friend
and
your
hated
enemy
at
the
exact
same
time.
For
precisely
that
reason,
so
I
understand
wholeheartedly
what
you're
referring
to
so
so,
but
I
I
think
in
terms
of
themes.
A
So
now
that
you
basically
have
that
templated
sql
runner
right
to
basically
allow
your
business
users
to
not
not
write
in
scala
right
and
also
be
able
to
go
ahead
and
just
write,
sql
statements,
then
there's
also
that
automatic
validation
which
you
we
had
talked
about
before
in
terms
of
the
customized
airflow
operations.
Can
you
provide
a
little
context
behind
that
again?
If
you
don't
mind.
C
C
It
would
use
python
necessary,
airflow
macros
to
just
replace
the
macros
we
had
in
ruby.
Everything
worked
out
fine
with
magic
of
regex
again,
so
we
now
had
python
airflow
ginger
two
temples,
and
it
would
run
that
so
no
problem.
Second,
we
would
have
spark
operator,
we
also
like
a
bunch
of
customization,
integrated
ways,
monitoring
system
etc.
So
we
didn't
use
any
out
of
the
box.
We
had
pi
spark,
which
did
a
whole
other
thing
which,
like
docker,
containers
and
stuff.
C
C
Second
thing
all
of
this,
but
prefix
migration.
So
migration
data
break
sql,
run
operator,
migration
by
spark
operator,
migration
spark
3
operator,
et
cetera
so,
and
those
who
do
what
I
describe
like
they
would
just
replace
so
in
in
spark
in
spark.
3
operator
is
harder
to
do
to
replace
the
outputs,
because
you
cannot
inspect
scala
and
see
what
are
the
outputs
and
we
had
to
force
our
users
to
do
like
the
thing
where
they
have
to
specify
this
is
the
output
and
we,
then
we
could
replace
it
and
for
spark
sql.
C
We
had
all
the
power
we
just
analyzed.
It
magically
replaced
it.
No
one
had
to
do
it
like
very
useful
and
yeah
by
spark.
They
probably
did
the
same
thing
as
sparks
yeah.
So
this
is
the
techno
piece,
also
a
bunch
of
custom
operators,
but
then
we
discovered
that
we
want
to
do
a
bunch
of
more
like
some
of
the
teams
raced
ahead.
So,
like
maybe
alex
we'll
talk
about
this
more,
but
at
the
beginning
we
thought,
okay,
we'll
do
it
layer
by
layer.
C
First,
the
tail
end
the
very
last
task
which
have
no
down
streams,
then,
after
that
next
layer,
the
connect
player,
but
something
said
like
no:
no,
no
we're
going
to
do
it
in
one
week
and
they
just
migrated
all
of
the
tasks
and
we're
like.
Oh
no,
we
couldn't
well,
it
doesn't
work
because
it
breaks
the
synchronization.
C
As
alex
said
we
as
soon
as
someone
writes
from
the
graded
side,
it
automatically
stops
synchronization
for
not
only
for
that
task
for
all
the
downstreams
which,
at
that
point,
weren't
migrated
yet
so
it
just
did
a
bunch
of
super
magic
like
it's
called
trigger.
Rule
customization.
So,
just
like
this
trigger
rule
would
look
what
kind
of
operators
it
had
downstream
and
like
analyzed
up
streams
and
decided.
C
Oh,
I
can
trigger
this
task
or
I
will
skip
it,
and
this
will
allow
a
bunch
of
teams
independently
to
raise
the
head
when
creating
the
tasks
and
and
only
like,
when
data
engineering
said
it's
ready,
we
would
switch
over.
I
guess
diagram
would
be
helpful
here.
We
hope
not
have
no
diagrams
just
use
your
imaginations,
I'm
sorry!
Actually
we
should
have
that
car.
A
C
So
yeah
so
like
trigger
rule
very,
very
cool
thing
and
yeah.
At
one
point
I
had
to
like
customize
like
to
allow
the
whole
deck
to
run,
but
also
not
run
at
the
same
time
and
just
be
successful
and
allow
retriggering,
because
we
had
like
we
had
like
10
teams
at
the
same
time
doing
their
own
thing
and
the
coordination
that
is
those
like
customization,
very
helpful,
and
now
we
don't
need
those
customization.
Yes,
okay,.
B
I
would
like
to
add
to
some
of
this
maybe
expand
on
some
of
these
topics.
So
to
start
with
our
custom
migration
operators
in
airflow,
they
did
multiple
things,
as
staff
mentioned.
First
thing
they
were
doing
is
looking
for
whatever
the
means
were.
We
were
finding
the
the
destination
where
they
were
writing
and
we
were
swapping
those
destinations
so
that
we
don't
accidentally
overwrite
the
data.
That's
been
data
synced
until
we're
ready
to
on
top
of
that,
those
migration
operators
were
actually
using
just
to
be
safe.
B
B
They
were
actually
writing.
At
the
end
of
the
run,
they
would
write
a
piece
of
configuration
somewhere
saying
I'm
migration
operator
x.
I
finished
my
run
for
date,
whatever
the
date
is,
and
I
my
my
output
tables
are
abc
right.
So,
at
the
end
of
the
entire
dagger
run,
we
have
another
dag
that
kicks
off
our
validation,
dag
and
the
first
thing
it
does.
It
goes
checks
this
configuration
and
says:
okay.
B
These
are
the
things
that
I
have
to
validate
for
today,
and
it
will
just
go
pick
up
and
do
those
violations
so,
like
all
of
this
was
automated,
where
users
just
had
to
do
the
migration
and
as
long
as
they're
using
the
right
airflow
operators,
all
of
the
safeguards
and
automatic
validations
were
provided
to
them
as
a
side
benefit
so
yeah.
This
was
pretty
good.
Another
thing
that
I
want
to
talk
about
here
is
alluded
to.
Is
it
wasn't
just
us
doing
the
migration
of
all
of
those
tasks?
B
There
was
a
bunch
of
different
teams.
We
we
tried
to
migrate
as
many
things
as
we
could
ourselves,
but
at
the
end
of
the
day,
not
everything
is
a
sql
template.
There
are
some
spark
jobs
with
lots
of
logic
in
them,
there
are
some
ruby
jobs
that
have
to
be
migrated
to
spark
jaws
to
which
also
have
a
bunch
of
logic
in
them,
and
we
didn't
feel
super
comfortable
taking
over
this
like
migration,
because
we
don't
really
understand
what
the
job
is
doing.
B
So
the
the
teams
that
originally
wrote
them
had
to
chip
in
and
do
their
work
and
it
very
quickly
became
a
nightmare
for
us
to
to
just
keep
track
of,
what's
being
migrated.
What's
where
and
we
realized
that
we
need
a
way
to
to
keep
track
of
all
of
this,
we
need
a
way
to
tell
teams
hey.
You
can
start
working
on
this
task,
because
this
is
the
task.
That's
blocking
a
lot
of
other
teams
from
doing
their
migration
and
we
used
the
most
powerful
ui
tool.
There
is
we.
B
B
There
is
you
have
to
do
a
bunch
of
api
calls
and
security
tokens.
By
the
end
of
the
day
we
were
able
to
bring
this
data
into
a
databricks
notebook.
We
were
able
to
parse
our
dags
in
both
legacy
system,
as
well
as
the
new
system,
so
we
have,
and
then
we
join
all
of
this
data
together,
break
it
up
by
teams.
Do
a
nice
bunch
of
pretty
graphs,
so
teams
can
see.
Where
are
they
at
how
many
tasks
they
have
remaining?
B
What's
whom
are
they
blocking?
What's
the
most
important
thing
to
work
on
next
and
stars
did
a
bunch
of
customizations
to
that
later.
It
also,
I
think
we
also
integrated
all
of
the
validation
statuses
for
all
of
those
tasks
in
that
dashboard,
so
it
quickly
became
from
it.
It
quickly
went
from
like
the
total
wild
wild
west
to
a
semi
wild
west.
So
it's
a
wild
west
with
a
bunch
of
pretty
graphs.
Basically.
A
You
had
to
make
it
so
that
way,
individual
teams
could
actually
go
ahead
and
do
the
migrations
without
actually
necessarily
having
like
a
full
cue
structured
flow
right
that
they
could
actually
figure
out
where
they
were,
and
then
it
actually
going
all
the
way
back.
It
started
with
not
just
the
templates
that
you
were
doing,
but
basically
building
these
customized
airflow
operations.
These
dags,
where
each
migration
task
itself
would
go
ahead
and
produce
a
bunch
of,
in
essence,
metadata
right
to
tell
you
what's
going
on
and
these
airflow
tasks.
A
That
would
then
read
that
metadata
to
validate
so
metameta
right
and
then
once
it
did
all
of
those
things.
Yes,
whether
it's
google
sheets
or
databricks,
notebooks
or
whatever
else,
but
what
ultimately
happened,
was
basically
these
beautiful
graphs
of
dags
that
tell
you
which
set
of
tasks
actually
were
migrated
and
then
for
any
team
they'd
be
able
to
go
ahead
and
figure
out
where
they
were
so
they
could
go
ahead
and
pseudo
independently
work
from
each
other.
Would
that
be
in
an
right
from
a
nutshell?
Or
am
I
missing
some
some
tasks
here.
B
In
a
nutshell,
yes,
I
mean
the
devil
is,
of
course,
isn't
in
the
details
and
some
things
did
break,
but
there
weren't
there
weren't
any
like
massive
breaks
for
us.
We
we,
as
you
mentioned
lots
of
metadata
and
really
safeguards,
was
self-service
with
safeguards
really
was
the
key
to
this
right
so
that
if
they
shoot
someone
in
the
foot,
it's
only
themselves
that
they
shouldn't
put
it
and
should
everyone
else
in
the
room.
A
Got
it
so
what's
interesting
here
is
that
you
know
the
discussion
we're
having
here
about
like
guard
rails,
for
example,
it's
very
micro
services-ish,
like
very
in
terms
of
thinking
we're
basically
like
independent
teams
working
from
each
other,
but
then
your
job
is
to
basically
put
these
guard
rails
up.
So
that
way
they
don't
go
completely.
A
You
know
off
kilter.
What
I'm
curious
about
is
that
from
your
team's
stocks,
whether
it's
yourself
or
alex
you're
yourself
are
from
your
backgrounds.
Were
you
originally
databased
people,
and
then
you
transitioned
to
microservices
and
big
data
are
where
you're
always
like
thinking
from
a
very
microservices
perspective.
B
So
in
my
past
I
had
quite
a
breath
of
experience
and
I
would
say
I've
seen
a
little
bit
of
everything
and
I
created
strong
opinions
on
on
different
approaches
and
yeah.
We
had
to
work
with
microservices
architecture,
probably
four
or
five
years
ago,
and
it's
but
it's
not
all
rows.
I
mean
it's
all
good,
but
it's
it's
hard
to
create
a
localized
development
environment
with
20
micro
services
running
trying
to
do
an
end-to-end
testing
but
anyways.
Yes,
we,
my
my
background,
has
been
in
classical
databases.
A
No,
no,
it
makes
sense.
I'm
I'm
actually
alex
you
and
me
you
and
I
actually
have
very
similar
backgrounds
from
that
standpoint.
So,
yes,
I
completely
grok
you
where,
like
I'm,
not
I'm
not
one
to
say
it's
all
about
microservices,
I'm
just
saying
like
it
seems
like
if
you
were
to
also
only
go
in
the
data
waste
world
and
you
were
trying
to
do
that,
we
would
actually
try
to
migrate
1300
tasks
all
at
once,
right
in
the
monolith
perspective,
which
er
that
probably
wouldn't
work
either
so.
B
This
is
actually
an
interesting
point,
because
we
one
of
the
things
that
we
were
looking
at
when
we
were
migrating
this
gigantic
dag
of
1500
minus
something
tasks
right
is
that
we
initially
looked
at
can.
Can
we
actually
break
this
thing
up
into
multiple
dags
because
it's
insane
to
try
to
keep
track
of
all
of
those
things
and
all
of
these
dependencies,
and
when
we
were
evaluating
that
in
an
original
system?
B
But
when
we
moved
at
the
end
of
the
day
when,
when
we
moved
to
our
new
platform,
we
were
able
to
slice
off
like
at
least
like
two
to
three
hundred
tasks
into
a
separate
dug
that
can
run
independently
and
because
now
all
of
these
tasks
are
running
much
faster
for
multiple
reasons.
We
we
we
can
afford
that.
We
don't
have
to
basically
rush
to
start
a
task.
We
can
choose
simplicity
of
maintenance
over
performance
here,
because
we
don't.
We
no
longer
have
those
like
very
slow
pipelines.
A
Oh,
that's
super
interesting.
Actually
I
want
to
dive
into
that
a
little
bit,
but
before
I
do
that,
stop
how
about
yourself
I'd
love
to
know
a
little
bit
about
your
background
in
relation
to
this.
You
know
the
trans,
this
transition,
like
you,
did
you
always
come
from
a
you,
know
more
applications
or
services
development
background.
Or
did
you
come
from
a
model
architecture
background
before
you
started
this.
C
I
worked
in
a
java
background
with
analyst
architecture
and
then
like
a
bunch
of
spring
boot
applications.
If
you
want
to
create
platform
to
test-
and
I
think
springboard
here-
yes,
it's
fine,
but
yeah
yeah.
I
had
my
there
here
and
there,
but
I
I
would
say
that
in
this
particular
instance,
we
had
no
choice
like
it
was.
C
Knowledge
was
distributed
so
widely
and,
as
alex
said,
we
couldn't
break
it
up
into
pieces
to
just
do
in
parallel
or
separately
with
other
teams
yeah,
we
had
to
rely
on
other
teams
because
of
the
we
didn't
have
all
the
knowledge
and
it
just
the
load
was
too
much
yeah
yeah
and
we
tr
when
I
was
thinking
about
breaking
up
our
big
kitchen
sink
deck
is
what
we
call
it
into
multiple
pieces
and
fun
fact.
Now
it's
so
large
github
refuses
to
index
it.
A
That's
actually
pretty
cool.
I
I
have
been
guilty
of
placing
very
large
files
in
the
github
for
various
different
reasons.
But
yes,
I
understand
exactly
what
you're
talking
about
so
in
terms
of
like
in
terms
of
trying
to
wrap
to
wrap
things
up
for
at
least
today's
session.
It
seems
to
me,
like
an
overriding
theme,
was
also
in
the
process
of
doing
all
these
migrations
you
kept
on.
A
It
was
very
difficult
to
make
things
simple,
but
in
the
process
of
making
things
simple,
you
could
actually
literally
choose
the
the
simplifying
operations
over
performance
because
you
actually
no
longer
had
slow
pipelines
in
the
process.
I'm
just
wondering.
Is
there
anything
else,
you'd
like
to
add
to
that
sort
of
that
theme
per
se,
then.
B
Theme,
I
would
say
theme
is
correct
where
we
were
trying
to
first
of
all
reduce
our
technical
footprint
right,
our
technology
footprint.
We
we
decided.
Okay,
we
want
to
run
everything
in
spark.
We
don't
want
to
have
hive
impala
spark
ruby,
you
name
it
right,
there's,
probably
something
that
I
forgot
to
mention
there.
We
we
didn't
want
that.
We
want
everything
in
spark
one
set
of
tools
to
monitor
one
set
of
logging
tools
etc.
A
B
So
a
couple
things
that,
like
maybe
just
call
out
some
some
things
that
we
didn't
have
to
solve
as
we
were
moving
over,
we
were
initially
originally
thinking
how
we're
going
to
solve
this,
and
then
we
didn't
have
to
so.
To
start
with,
we
have
lots
of
jobs
that
are
templated
sql
and
when
we
were
starting,
the
migration
spark
3
wasn't
around.
B
So
we
were
looking
at
them
and
saying,
let's
analyze
their
runtimes,
the
ones
that
don't
take
much
time
to
run
will
keep
them,
as
is
anything
else.
That's
taking
more
than
x
minutes
to
run
we'll
have
to
go
dig
into
them.
We
might
need
to
convert
them
to
spark
jobs,
do
a
bunch
of
optimizations
here
and
there
and
then
spark
3
comes
out,
and
it's
like
yay.
We
can
just
turn
on
automatic
optimizations
and
just
throw
it
at
it.
B
The
next
problem
we
had
to
solve
was
let's
say
we
talked
about
it,
a
lot
optimizing
our
outputs
right
in
the
previous
session,
but
now
we
can
just
turn
on
hey
automatically
optimize
our
outputs
and
we
don't
have
to
solve
this
problem
so
that
that
allowed
us
to
keep
pretty
much
almost
100
of
those
sql
templates
as
sql
templates.
Without
introducing
this,
let's
make
it
now
a
more
complex
spark
job
with
lots
of
micro
optimizations
within
it
right
so
and
now
we
don't
have
to
maintain
it.
B
B
Another
thing
that
we
achieved
with
this
migration-
we
talked
about
it
in
the
previous
session,
is
workload
isolation.
So
because
our
workloads
are
isolated,
we
basically
schedule
each
job
on
its
own
cluster.
We
don't
have
this
fighting
over
network,
I
o
between
jobs.
We
don't.
We
no
longer
have
to
limit
the
concurrency
of
our
tasks
in
our
legacy
system.
B
Let's
make
them
simpler
right,
so
we
can
actually
it's
also
as
a
company
we're
growing
and
maturing.
It's
no
longer
hey,
there's
these
10
guys
that
know
everything
about
every
piece
of
the
system.
It's
now
hey.
We
have
this
hundreds
of
users
now
using
this
system
and
let's
partition
the
work.
Let's
partition
the
knowledge,
but
make
sure
that
we
have
tools
and
guards
in
place
to
make
sure
it
still
works
and
they
don't
interfere
with
each
other.