►
Description
Join us for part three of a three part tech talk series: Upgrading from legacy to the cloud with Scribd. This is the final session, Moving Ad Hoc Users to the Cloud. Alexander Kushnir, R. Tyler Croy, and Hamilton Hord from Scribd discuss with Denny Lee from Databricks the technical and business issues around moving ad-hoc jobs to the cloud as part of Scribd’s migration from legacy environments to the cloud.
In this session, we dive into a variety of topics including exploratory non-dev use cases, how Scribd moved development into Databricks, model training use cases, and shared cluster resources. Listen to how Scribd engineers used Delta Lake to solve their production distributed cloud data issues.
Part One Recording: Replicating Data to the Cloud Recording - https://youtu.be/vGv6AcPp7Zs
Part Two Recording: Moving Batch Jobs over to the Cloud - https://youtu.be/siVvtalssrI Databricks is proud to announce that Gartner has named us a Leader in both the 2021 Magic Quadrant for Cloud Database Management Systems and the 2021 Magic Quadrant for Data Science and Machine Learning Platforms. Download the reports here. https://databricks.com/databricks-named-leader-by-gartner
A
Third
session
of
the
series,
I'm
concerning
upgrading
from
legacy
to
the
cloud
with
scribd
today's
session
is
actually
going
to
be
focusing
on
moving
ad
hoc
users
to
the
cloud.
My
name
is
denny
lee,
I'm
your
host
for
today's
session,
I'm
a
developer
advocate
at
databricks.
Prior
to
this
I
used
to
be
at
microsoft.
I
happened
also
worked
on
the
project
isotope,
where
we
moved
our
big
data
platform
to
get
hadoop
on
windows
and
azure.
A
A
B
I'm
going
to
give
you
a
little
bit
more
history
of
myself
that
I
did
in
the
prior
two
sessions.
Like
a
true
high
quality
data
engineer.
My
first
software
engineering
job
was
writing
reports
in
microsoft
access.
So
this
is
where
I
started
and
slowly
moved
my
way
up
at
some
point.
They
let
me
touch
a
real
database,
which
was
microsoft,
sql
server
and
believe
it
or
not.
My
first
cloud
was
azure
and
hd
insights
was
my
first
big
data
platform
that
I
got
my
hands
on
so
danny.
There
you
go.
It's
all
comes
back
here.
A
C
A
D
A
D
A
Oh,
that
sounds
pretty
cool.
Actually
that
could
be
its
own
recording
session.
But
let's,
let's
not
digress
too
much
alex.
Why
don't
we
start
by
reminding
some
of
the
viewers
here
a
little
bit
about
the
legacy
migration
to
the
cloud
and
moving
bash
jobs
to
the
cloud
as
part
of
the
previous
two
sessions
just
to
remind
viewers
what
it
took
for
you
to
go
from
your
on-prem
systems
to
the
cloud
that
you're
currently
working
with
right
now.
B
B
A
Now
we
actually
want
to
discuss
a
little
bit
about
moving
ad-hoc
users
to
the
cloud
and
there's
a
lot
of
really
interesting
things
like
whether
it's
the
exploratory
non-developer
uses
cases
how
you,
how
scrip
move
development
into
the
cloud
model,
training,
use
cases
and
shared
cluster
resources.
So,
in
order
for
us
to
kick
start
it
I'm
going
to
just
go
through
each
of
these
four
themes
and
then
tyler,
maybe
I'll
start
with
you
a
little
bit
about
the
exploratory
non-development
use
cases,
since
I
think
that's
a
good
way
to
start
off
sure.
C
One
of
the
things
that
I'd
like
to
point
out
with
folks
when
we
talk
about
the
data
use
cases
at
scribd
is
that
data
is
sort
of
key
to
what
makes
scribd
as
a
business
work.
You
know
if
you've,
if
you've
ever
worked
with
scribd
or
scene
script
in
the
wild,
there's
user,
uploaded
documents,
there's
books,
there's
an
incredible
amount
of
content,
and
as
such,
we
do
a
lot
of
interesting
and
novel
things
with
that
content.
C
So
data
data
and
access
to
data
is
a
pretty
key
aspect
for
a
lot
of
people's
jobs
at
scribd
when
we're
in
our
I'll
start
with
the
on
premise
our
legacy
set
up
and
and
what
that
entailed
for
people
you
know
in
in
the
classic
hadoop
environment,
pretty
much
all
we
could
offer
people
at
the
time
was.
You
know,
access
to
spark
shells
which
alex
will
talk
a
little
bit
about
later
or
for
most
of
the
people
who
wouldn't
have
called
themselves
a
developer
access
to
hue.
C
You
know,
which
is,
is
basically
like
php,
my
admin
on
top
of
hive
and
for
a
lot
of
people.
What
ended
up
happening
was
data
was
impenetrable.
So
we
had
at
the
time
our
now
called
applied
research
team.
Then
it
was
just
called.
The
data
team
would
end
up
servicing
a
lot
of
requests
from
people
around
the
organization
for
like
I
need
to
know,
answers
to
this
question
with
data,
but
I
don't
know
how
to
get
that
data.
C
I
don't
know
where
to
get
that
data.
I
don't
know
what
tables
that
might
exist
in.
I
can't
I
don't
have
the
wherewithal
to
start
there.
So
please
just
go
find
my
answer
for
me,
which,
as
you
might
expect,
is
a
very,
very
inefficient
way
of
doing
things
we
also
technically.
We
also
had
tableau
running
around,
but
tableau
is
just
as
impenetrable
for
a
number
of
people
as
he
was
so
we
don't
need
to
talk
about
that
either.
C
What
ended
up
happening,
which
was
really
fascinating
to
me
to
see,
is
once
we
opened
up,
like
you
know,
moving
moving
our
data,
as
alex
referred
to
in
previous
sessions,
setting
up
that
data.
Sync
actually
allowed
people
to
go
start
explore
data
and
experiment
with
data
in
our
data
warehouse
or
the
user
generated
content
that
we
have
stored
in
in
s3
in
ways
that
we
didn't
actually
expect.
C
Probably
my
favorite
use
case
that
I've
seen
that
you
know
we
actually
didn't
account
for
previously
at
all.
Is
developers
or
product
managers
setting
up
notebooks
while
they're
developing
a
feature
that
they're
then
going
to
run
immediately
after
they
deploy
something,
so
they
can
actually
create
these
notebooks
that
define
their
acceptance
criteria
and
once
something
is
shipped
then
they're
just
going
in
you
know,
once
data
starts
flowing
in
whether
that's
through
batch
workloads
or
streaming
workloads,
then
they're
just
running
you
know.
Did
this
feature
work
notebook,
the
other
really
really.
C
They
would
be
getting
data
that
they
could
work
with,
and
once
we
started
to
have
that
data
available.
My
other,
like
I,
was
blown
away
when
I
saw
this
because
I
didn't
even
think
about
it.
I
saw
developers
as
well
using
using
notebooks
and
querying
our
data
to
do
production
incident
triage,
which
was
something
that
was
like
galaxy
brain
like
I
was
so
amazed
when
I
saw
it
because
what
ended
up
happening
is
we
had
this
live
incident
where
we
had
to
find
you
know
some.
C
They
were
pulling
together
all
of
this
data
in
one
notebook
to
identify
who
are
the
users
that
were
affected.
How
were
they
affected
and
actually
helped
in
a
lot
of
ways
identifying
what
the
underlying
bug
was,
and
all
of
that
sort
of
that
this
sort
of,
like
amazing,
secondary
benefits
that
we
we
realized
by
moving
this
ad
hoc
workloads
into
the
cloud
strictly,
came,
in
my
opinion,
from
making
data
accessible
and
easy
for
people
to
discover
and
then
letting
them
figure
out
what
they
want
to
do
with
it.
C
The
pandora's
box
has
been
open,
like
everybody,
everybody
is
already
benefiting
from
all
of
this
data,
where
they
pretty
much
before
had
to
go
to
individuals
in
the
office.
You
know
these
were
the
olden
times
and
say
I
need
I
need.
I
need
a
question
answered.
Please
can
you
go
consult
with
the
data
and
then
come
back
to
me
now.
People
are
just
sort
of
self-serviced
and
going
and
figuring
out
what
they
need
to
figure
out
to
do
their
jobs,
which
is
fantastic.
A
So
one
thing
I
did
want
to
focus
a
little
bit
about
that
you
sort
of
alluded
to,
but
I
wanted
to
bring
to
the
forefront
and
with
tyler
whether
you
want
to
go
into
it
or
or
hamilton.
You
want
to
go
into
the
let's
talk
a
little
bit
about
like
those
streaming
use
cases
and
also
about
like
the
ability
to
work
with
logs.
A
I
mean
you've
talked
about
that
use
case
where,
in
essence,
your
product
managers
are
actually
using
the
data
to
understand
the
features
that's
great,
but
then
there
seems
to
be
that
implication
that
they're
able
to
build
those
notebooks
and
start
accessing
the
data,
mainly
because
they
actually
had
immediate
fidelity
or
immediate
access
to
the
data.
So
why
don't?
We
talk
a
little
bit
about
that?
From
that
perspective,
yeah.
C
This
actually
comes
down
to
some
of
the
the
road
map
that
core
platform,
as
a
team
was
working
on
in
2020,
like
we
really
wanted
to
try
to
make
as
many
data
sets
that
were
available
as
streams
or
produces
streams.
I
should
say
available
as
streams
so
in
in
in
the
before
times
we
had
like
data
would
flow
into
kafka
and
then
hamilton,
I
think
we
were
just.
Was
it
flume
touching
things
once
a
day?
I
don't
actually
remember
how
flume
worked.
D
Flume
would
occasionally
batch
files
into
hdfs
and
then
one
of
our
one
of
the
first
tasks
that
would
kick
off
on
the
nightly
run
would
taking
that
data
and
putting
that
into
hive
to
where
it's
then
accessible
by
the
rest
of
the
batch
jobs.
So
even
the
most
raw
of
rawest
of
data
wouldn't
be
available
until
the
nightly
started
kicking
off.
Even
though
it's
written
relatively
often.
C
C
C
C
We
basically
come
back
and
we
run
a
batch
optimize
on
the
previous
day's
partition
and
so
like
there's
lots
of
delta
magic
going
on
in
these
tables.
It's
lots
of
fun.
But
that
means
that
you've
got
really
high
performance
queries
on
older,
older
partitions
in
the
table,
but
you
still
have
effectively
up
to
the
minute
updates
in
that
table
and
that's
the
thing
when
we
talk
about
streaming
and
sort
of
really
empowering
these
ad
hoc
users
or
exploratory
users.
C
A
And
that
actually
makes
a
lot
of
sense
right,
because
one
of
the
key
reasons
spark
had
built
structure
streaming
in
the
first
place.
You
know
if
I
was
to
take
the
more
marketing
aspect
of
it
was
things
of
saying
we
don't
you
don't
no
longer
had
to
think
about
streaming
in
order
to
be
able
to
have
streaming
right,
the
the
that's
why
we
built
structure
streaming
to
have
the
structured
context
and
then
with
delta,
that
took
that
one
step
further,
because
you
could
actually
start.
A
Finally,
decoupling
latency
from
the
business
logic
of
actually
running
queries
against
that
data.
So
your
users,
your
product
manager,
in
this
case
right,
could
go
ahead
and
run
the
same
query
whether
it
was
meant
to
be
a
static
table
that
was
six
hours
later
or
it
was
a
dynamic
table
that
was
constantly.
That
constantly
was
being
updated
with
new
data
due
to
a
streaming
process.
So
that's
pretty
cool
because
now
you're
just
so
showcasing
that
in
action.
Basically.
C
So
one
of
the
things
I
don't
want
us
to
get
off
on
a
tangent
here,
but
one
of
the
things
for
people
to
check
out
if
they're,
really
keen
on
streaming
with
delta
lake
delta
tables
can
be
streaming
sinks.
So
we
we
sync
data
into
it,
but
you
can
also
treat
them
as
a
streaming
source
which
is
really
powerful
for
a
lot
of
our.
You
know,
batch
or
other
live
streaming
workloads
where
we
will
stream
into
one
table
from
kafka.
C
Basically,
and
then
data
from
there
will
get
picked
up
by
another
stream,
just
looking
at
delta,
so
it
actually,
it
simplified
a
lot
of
our
streaming
topologies.
Just
by
saying
all
right
once
it
hits
the
you
know
the
or
initial
table,
then
we
can
just
use
that
as
the
streaming
source
for
anything
downstream
super
cool
use
case,
but
I
don't
want
to
get
too
much
off
of
our
ad
hoc
topic
with
it.
A
No
problem
at
all,
but
since
you,
since
you
did
open
the
door,
let
me
actually
call
it
two
things.
One
is
that
delta
lake
0.8
actually
includes
the
ability
to
not
just
stream
from
using
as
a
stream
source,
but
also
stream
from
a
particular
time
time,
travel
version
or
version
of
the
data.
So
that
way
you
can
go
back
to.
A
C
A
No
in
this
case,
though,
you've
heard
definitely
heard
me
talk
about
that
in
the
covet.
19
said
lectures,
but
no
in
this
case
change
data
capture.
You're
right.
I
should
have
definitely
called
that
out.
Thank
you
very
much
all
right.
Let's
go
now,
I'm
just
as
guilty
as
you
for
taking
us
off
course.
So
let's
go
back
to
our
course,
and
you
know
what
we'll
do
we're
going
to
bring
it
back
to
alex,
because
you
know
alex
actually
keeps
us.
A
It
keeps
us
honest
here,
okay,
so
let's
talk
a
little
bit
about
how
you
had
the
script
had
basically
moved
the
development
into
the
cloud
environment
and
how
to
move
to
databricks
and
let's
start
specifically
about
the
spark
shell
users,
because
I
think
we
had
chatted
about
this
in
the
past.
But
why
don't
we
elucidate
a
little
bit
about
that
on
on
those
spark
shell
users?
What
was
going
on
before
and
now?
What's
it
like
working
with
them
now.
B
Sure
so
spark
shell
is
a
great
tool
right
and
if
all
you're
trying
to
do
is
write
some
spark
in
the
shell
and
that's
about
it,
but
unfortunately
for
us
we
had
too
many
spark
shell
users
and
the
reason
for
that
is
really.
We
had
two
interfaces
to
our
data
for
end
users.
We
had
spark
shell
and
hue
and
our
business
analytics
users.
They
would
prefer
hue
over
spark
shell.
B
I
can't
blame
them
right
for
that,
but
a
hue
running
hive
wasn't
too
fast,
so
we
said:
okay,
let's
have
impala
and
sure
impala
is
kind
of
fast.
But
oh,
why
is
it
giving
me
this
weird
data
that
doesn't
actually
match
my
hive
data?
Well,
because
you
got
to
keep
the
catalogs
and
sync,
you
have
all
sorts
of
things
you
have
to
do
to
to
get
it
to
work
right.
B
C
B
B
Oh
yeah,
it
sounds
like
you,
oh
here
it
looks
like
you're
doing
something
really
weird
and
oh
yeah.
Try
this
instead,
oh,
let
me,
oh
man,
how
do
I
paste
this
big
text
into
spark
back
into
spark
shell?
Oh
man,
this
is
horrible.
Let
me
go
learn
to
do
that
and
at
the
end
of
the
day
it
works.
Oh,
but
now
I'm
having
a
performance
issue
all
right.
Let
me
go
into
my
resource
manager.
Try
to
find
your
spark
shell
here
now
I'm
going
to
get
into
your
spark
ui
and
say:
hey.
B
Let
me
get
into
this
spark
ui
and
look
at
it.
Oh
okay,
yes
looks
like
you're
having
some
kind
of
skew
problem
here.
Let
me
go
research,
your
data,
a
little
bit
where,
where
can
I
research
your
data?
Oh
group,
that's
right!
In
a
spark
shell!
Let
me
go
try
to
get
to
pull
some
basically
data
distribution
metrics
from
your
data
in
anyway.
So
this
is
this
is
what
we
were
doing
and.
B
So
yeah,
so
at
the
end
of
the
day,
it's
it
was
very
productive
and
when
actually
there
was
a
conversation
happening
somewhere
when
we
were
just
taking
on
this
before,
I
think
we
even
took
on
this
big
migration
project
to
the
cloud.
It
was
like:
hey
guys,
how
can
we
make
our
users
more
productive?
Can
we
get
them
some?
Can
we
get
them
all
licensing
for
some
kind
of
like
development
tools,
so
they
can,
and
my
answer
was
like
we
should
at
least
try
some
kind
of
notebook
solution
for
them.
B
Guys,
like
I
mean
I
used
zeppelin
notebooks
before
and
that's
not
stellar,
but
even
that
is
a
much
more
productive
development
process
than
than
running
around
the
spark
shells
right
so
and
of
course,
now
we
bring
into
today
when
those
users
saw
databricks
notebooks,
it's
like.
Oh,
it's
like
hey.
Why,
like
you,
can
do
that
right?
B
Why
were
we
doing
this
other
thing
before?
So,
of
course
it
took
us
some
time
to
get.
We
talked
about
data
sync
to
get
data
up
into
the
cloud
to
enable
those
use
cases,
but
once
it
was
there
all
of
a
sudden,
our
developers
are
loving
it
because
they
no
longer
have
to
use
spark
shell,
even
though
you
still
can
in
when
you're
starting
up
your
databricks
clusters.
You
still
can
set
up
your
spark
shell
and
for
the
few
users
that
we
have
that
really
miss
it.
I
tell
them
hey,
you
could
be
go.
B
Do
it
it's
your
thing,
but
in
any
event,
it
actually
also
changed
the
way
we
communicate
because
now,
instead
of
having
this
long
conversation
in
a
slack
with
lots
of
copy
pasting,
formatting
and
staring
at
the
code,
you
just
get
a
notebook
link,
saying
hey
I'm
having
this
problem
here.
Can
you
look
at
this
and
in
the
notebook
I
see,
I
can
see
not
just
the
line
of
code.
That's
causing
the
problem.
I
can
see
the
entire
context
right.
I
see
what
they
did
before.
B
I
can
query
the
current
state
of
of
their
o,
o
o
of
where
are
they
in
right
and
another
good
thing
is
that
I
can
actually
see
the
spark
ui
right
there
and
then
without
having
to
go
dig
for
those
things
and
one
thing
that
I've
been
encouraging
our
developers
to
use
a
lot.
Is
that
hey
guys?
You
now
actually
have
this
ganglia
right
here
right
next
to
you
go
use
it.
B
B
You
still
have
to
later
on
run
it
in
the
real
environment
to
optimize
it
for
the
volume
of
data
that
you
have
for
any
kind
of
data
characteristics
such
as
sku,
et
cetera
that
you
have
to
deal
with
right.
So
our
now
we
have
that
ability.
We
have
not
just
shell
to
do
this.
We
have
very
powerful
tools
to
do
that.
A
A
This
is
very
much
the
context
of
like
you
know
the
embedded,
spark,
ui
and
ganglia
all
within
the
databricks
notebook
environment.
So
this
allows
users
to
basically
not
need
to
pull
up
multiple
different
environments
in
order
to
be
able
to
see
the
same
information
just
as
alex
so
loquaciously
described
right.
Instead,
it's
just
all
in
this
one
little
place,
so
it's
really
easy
to
find,
and
just
as
a
quick
tip
for
all
you
folks
out,
there
don't
forget
to
tag
your
resources
as
well,
whether
it's
on
the
database
side
or
whichever
cloud
environment.
A
So
that
way,
as
alex
called
out,
if
you
do
need
to
slap
people
on
the
wrist,
you
have
a
better
way
to
figure
that
out
all
right,
perfect.
Well,
I
mean
this
just
really
did
a
great
job
in
terms
of
describing
the
jobs,
but
then
can
we
discuss
or
just
talk
a
little
bit
about
more
about
now
that
you've
moved
the
development
into
the
cloud.
Slash
database
environment
and
we've
also
discussed
a
little
bit
about
those
exploratory
use
cases.
A
What
are
some
of
the
other
use
cases
that
we
can
probably
dive
into,
and
I
think
the
first
thing
that
we
were
thinking
about
was
a
little
bit
about
the
model
trend,
use
cases,
and
so
can
we
talk
about
how
that
worked,
whether
it's
you
know
tyler
or
alex
or
hamilton?
You
know,
I'm
not
not
sure
who
would
like
to
dive
into
this.
First.
Actually.
C
For
our
old
model
training,
so
machine
learning
at
scribd
is
is
fairly
tenured.
I
guess
would
be
the
word.
I
don't
have
the
vocabulary
that
you
do
mr
liquaciously
over
there,
but
we've
been
we've
been
developing
models
for
probably
three
or
four
years
now
that
had
to
be
trained
on
different
sets
of
data
and
then
re
you
know
launched
into
production
and
things
like
that.
C
C
B
C
I
want
hamilton
to
talk
about
how
some
of
our
clusters
for
these
use
case,
these
ad
hoc
use
cases
are
set
up,
but
what
we've
ended
up
doing
is
extending
cluster
create
permissions
to
a
number
of
the
the
individuals
on,
say,
the
recommendations,
team
or
the
applied
research
team
or
the
search
team
that
need
to
do
their
their
model,
training
and
they'll.
Just
you,
you
know
through
databricks,
literally
spin,
you
know
whatever
beefy
beefy
stuff
up
that
they
need
in
in
databricks
and
then
they'll
start
training,
there's
definitely
some
things
that
can
be.
C
Let's
call
it
interesting
with
training
on
data
bricks.
In
that
a
lot
of
the
types
of
models
that
folks
are
training.
They
can't
take
advantage
of
the
distributed
nature
of
a
spark
cluster,
but
they
still
need
spark
to
access
their
delta
tables
where
they're
sourcing,
their
training
set
from.
So
when
we're
looking
through
the
clusters
list,
we'll
see
these
really
really
like
oversized
big
giant
big
beefy
truck
door
arm
type
instances
where
it's
like
the
biggest
driver
instance.
C
You
can
imagine,
and
then
like
this
little
bitty
worker
instance
off
to
the
side,
because
they're
just
spinning
up,
you
know
a
big
massive
driver
instance
to
do
their
training
on
we're,
hoping
we're
hoping
to
find
a
little
bit
better
of
a
way
to
do
that.
But
right
now
it's
pretty
much.
You
know
using
data
bricks
as
as
our
portal
into
delta
lake,
but
also
just
to
give
us
on-demand
gpu
instances
that
are,
you
know,
tagged,
have
the
right
profiles
associated
and
all
the
other
access
controls
that
we
need
for
that.
A
C
Okay
and
if,
if
you've,
if
you've
ever
read
the
children's
book,
if
you
give
a
mouse
a
cookie,
it
is
a
very
good
parable
of
building
a
data
platform
for
a
data
focused
business.
In
that
you
give
them
one
thing
and
they
want
another,
and
then
they
want
another
and
then
one
another,
and
so
what
we've
actually
found
is
just
about
every
every
model,
training
framework
that
exists.
C
I
could
probably
go
find
that
internally,
because
the
way
that
a
lot
of
the
the
machine
learning
developers
work
is
they'll,
try
a
couple
of
different
frameworks
and
maybe
build
off
some
pre-existing
models
and
they'll
experiment
until
they
get
something
that
looks
like
what
they
want
and
that
we
we
certainly
want
to
enable.
But
what
that
means
for
us
from
a
like
delivering
tools
and
platform
standpoint.
Is
we
can't
just
say
like?
Oh
here's,
the
supported
tool
chain
for
you
to
go
with.
C
It
has
to
be
bring
your
own
tool
chain
just
bring
whatever
you
need,
and
so
we
have.
You
know
part
of
the
reason
why
we
needed
to
grant
these
users
the
ability
to
launch
clusters
is
so
they
could
also
install
their
own
libraries
into
these
clusters,
and
you
know
bring
whatever
tools
that
they
need
without
coming
to
talk
to
us
about
it.
A
Right
right
got
it,
so
it's
from
your
perspective,
it's
very
much
the
concept
of
self-service
machine
learning
in
essence
right.
The
ability
for
them
to
do
whatever
they
need
to
take
it
doesn't
doesn't
constantly
involve
basically
pinging.
You
guys
here
on
every
single
potential
library
that
they
want
to
go
play
with.
Basically,.
C
A
Actually,
that's
a
great
segue,
so
hamilton.
Yes,
we
were
going
to
talk
about
this
anyways,
which
was
about
how
do
you
manage
those
shared
cluster
resources?
So
this
is
an
excellent
segue
to
in
terms
of,
for
example,
the
ability
for
your
machine
learning
engineers
or
data
scientists
to
be
able
to
go
ahead
and
actually
generate
or
build
up
whatever
cluster
resources
that
they
need
so
yeah.
How
do
you
manage
it?
How
do
you
do
it?
I
also
alluded
a
little
bit
about
tagging,
so
yeah
by
all
means
please
dive
into
it.
Sure.
D
I
want
to
start
off
with,
like
one
of
our
biggest
premises
that
we
were
focused
on
building
around
is
the
sanctity
of
what
we
consider
our
prod
warehouse
so
like
our
the
s3
bucket
and
glue
catalog
that
we
use
to
store
what
we
consider.
Production
data
has
very
limited
right,
right,
access,
extremely
limited,
only
accessible
via
various
different,
automated
jobs
or
the
extreme
like
need
to
get
in
there
with
a
manual
touch
every
now
and
again
for
an
incident
or
something.
D
But
through
that
we
have
a
number
of
different
permissions
and
stuff
set
up
so
that
different
levels
of
access
to
that
that
data,
as
well
as
different
access
to
other
catalogs,
can
be
set
up.
So
that
developers
can
go
in
and
say
be
able
to
write
to
a
table
without
any
chance
of
writing
to
production
warehouse,
but
still
be
able
to
use
the
production.
Warehouse
data.
D
D
D
A
Got
it
that
makes
a
ton
of
sense
so
then,
you've
you've
mentioned
the
process
by
which
you've
given
in
essence,
everybody
access
to
the
shared
clusters
but
or
shared
resources.
Excuse
me,
but
then
I
guess
we're
going
a
little
backwards
here,
pun
intended.
What
do
we
talk
about?
What
was
it
like
before
that,
though,
right
like?
In
other
words,
it
seemed
very
straightforward
to
be
able
to
talk
about
like
okay.
Well,
now
everybody
can
they
can
access
the
shared
clusters.
We
have
the
right
permissions
right
tagging.
A
They
couldn't,
for
example,
share
the
results
of
their
data
or
they
had
a
hard
time
at
least
doing
it,
or
they
had
a
hard
time.
Having
conversations
about
it
because
they
weren't
using
a
shared
environment
and
so
forth
and
so
forth,
I
guess
if
I
was
to
talk
about
a
theme,
maybe
the
collaboration
aspect
of
things.
Gotcha.
D
So
the
big
collaboration
is
the
ability
to
have
like
those
notebooks
and
be
able
to
share
like
hey.
This
is
what
I
worked
on.
This
is
how
I
got
it,
and
here
are
the
results
from
it.
Just
in
like
a
url
versus
hey
query,
this
table
and
you'll
see
what
what
I
mean-
probably
maybe
additionally
like
in
our
hadoop
environment.
It
was
a
little
bit
of
the
wild
west
with
like
tables
and
stuff
people
all
were
using
the
same
metastore
catalog
there.
D
So
if
someone
was
working
on
a
table
that
they
needed
to
develop
and
while
two
two
people
could
work
on
the
same,
like
output
table
that
someone
was
developing
right,
like
I
couldn't
say,
write
a
table
with
one
of
my
jobs
and
be
drop
that
randomly
if
someone
else
was
needing
that
data
for
something,
so
it
was
very
difficult
to
kind
of
manage
all
these
spinning
plates
of
who
needed
access
to
what
table.
Hey.
Is
anyone
working
with
this?
No
okay?
I
can
go
ahead
and
drop
and
change
that.
D
Additionally,
in
the
hadoop
land,
we
were
limited
by
the
size
of
our
cluster.
So
we,
if
some
person,
wanted
to
hop
on
and
run
this
massive
like
query
and
use
like
60
of
the
cluster.
Well,
suddenly,
now
we're
getting
a
bunch
of
messages.
Saying
hey:
is
the
cluster
running
slow
for
anyone
else?
We
go
in
and
we
see
joe
schmo
running
like
this
cluster
util
90
clutch
cluster
utilization
thing
just
because
they
forgot
to
partition,
use
a
partition,
predicate
and
now
they're
creative
against
the
entire
historical
data,
causing
all
sorts
of
havoc
there.
D
D
Thing,
oh,
you
would
have
no
idea
how
many
incidents
were
caused
by
people
just
querying
a
lot
of
data,
but
by
allowing
some
of
our
more
power
users
the
ability
to
like
create
clusters
and
with
the
just
the
set
permissions
we
want
with
proper
tagging,
we
can
just
say
just
spin
up
what
you
need
go
wild.
You
won't
affect
our
general
users.
You
won't
affect
any
of
our
nightly
loads.
You
won't
affect
any
of
our
actual
other
production
use
cases.
You
can
just
go
crazy.
A
No
that
that's,
I
actually
feel
your
pain
completely
on
that
one.
I
have
so
had
the
environments
where,
when
I
set
up
my
own
hadoop
environments
and
what
ended
up
happening
was
this
gigantic
one
person
would
run
an
errant
query
that
would
basically
drain
every
single
resource
and
then
kick
every
single
other
person
out
or
prevent
them
from
even
getting
their
results,
let
alone
anything
else.
So
yes,
I.
I
definitely
feel
your
pain
on
this
one.
A
So
now
we've
talked
a
little
bit
at
the
past,
obviously,
and
let's,
let's
go
back
a
little
bit
more
to
the
toward
the
future
for
the
forward
or
to
the
future
right.
It's
my
understanding,
though
you
know
that
you've
actually
done
a
lot
of
terraforming.
Actually
so
that
way
for
the,
for
you
be
able
to
manage
all
of
these
different
resources
together.
A
So
that
way,
it's
possible
for
you
to
be
able
to
handle
this
idea
of
self-service
yet
also
be
shared
at
the
exact
same
time
like
this
idea
that
you
can
collaborate
together,
but
without
worrying
that,
or
at
least
minimizing
the
chance
that
you
take
out
everybody
in
the
process.
So
I'm
just
curious
well
for
those
who
may
not
know
what
is
you
know
terraforming
from
our
perspective
here
and
then
why
don't
you
describe
a
little
bit
about
the
environments
that
you've
done
for
databricks
and
the
cloud
for
that
matter?.
B
D
D
So
what
we
do
is
we
use
terraform
to
set
up
like
all
of
the
different
users
and
profiles
and
other
things
that
are
available
for
use
to
access
the
our
production
warehouse
and
then
also
with
terraform.
We
are
able
to
inject
those
into
data
bricks
for
users
to
use
there.
So
basically
I
don't
have
to
know
all
the
specific
role.
D
D
And
if
there's
any
sort
of
change
that's
needed,
we
have
a
git
repository
with
this
terraform
that
we
can
say.
Oh
here's,
when
this
line
got
changed.
This
is
who
changed
it
and
why
et
cetera
et
cetera,
and
we
can
go
through
the
history
of
the
entire
hoop.
Instead
of
going
wait
who
deleted
this
role,
we
can
just
go
into
the
code
and
see.
Oh,
this
person
deleted
this
for
x
reason
or
oh.
That
was
a
mistake.
Let's
undo
that
and
roll
it
back.
A
A
That
now
actually
is
in
fact
data
or
at
least
configuration
data
such
that
you
can
go
ahead
and
have
time
travel
on
its
own
in
essence,
code
in
essence,
we're
treating
it
more
closer
to
code
now,
so
that
way,
we're
able
to
roll
back
we're
able
to
deploy
infrastructure
from
a
cli,
as
opposed
to
you,
know
busy
clicking
on
20
000
buttons.
To
get
to
your
point
so
perfect,.
A
All
right,
this
has
been
really
interesting
and
I
did
want
to
end
today's
session
actually
to
talk
a
little
bit
about
what
are
some
gotchas.
You
know
from
you
know
we
for
the
last
three
sessions.
The
last
three
sessions,
including
this
one,
we've
talked
about
all
the
awesome
benefits
of
like
moving
to
the
cloud,
the
getting
the
batch
jobs,
the
cloud,
the
the
process.
Script
went
ahead
and
did
in
order
to
be
able
to
do
it.
But
what
are
some
gotchas
that
we
might
have
missed
that
I'd
that
you'd
like
to
call
out?
C
The
cloud
is
utilization
based
billing,
and
so
your
users,
you,
you
may
not
realize
it
when
you
enable
them,
but
you're,
actually
giving
your
users
each
a
little
bit
of
the
credit
card
that
they
can
just
go,
spend
and
spend
and
spend,
and
so
what
we've
done.
There
are
cluster
policies
which
we
we've
explored,
but
we
haven't
actually
set
in
action
yet
and
that's
a
way
to
to
help
manage
this.
To
a
certain
extent,
you
referenced
tagging
a
bit
earlier.
C
We
don't
have
there
are
things
on
the
aws
side
that
you
can
make
basically
make
tagging
a
requirement
or
certain
tags
or
requirement.
I
don't
remember
if
we
actually
roll
those
out
or
not.
We
do
have
a
pretty
strong
convention
of
tagging
things
so
that
one
isn't
shaking
as
now.
No,
no,
no,
we
didn't
roll
that
out
we're
we've.
Definitely,
we've
definitely
extended
a
lot
of
trust
to
people
in
the
organization
and
we're
we're
still
a
relatively
small
team,
so
we
we
can
certainly
afford
to
extend
some
of
that
trust.
C
Let's
call
it
intermediate
to
advanced
in
terms
of
skill
set.
We've
now
lowered
the
bar,
but
that
means
that
there's
there's
a
support
burden,
which
is
not
necessarily
a
bad
thing,
but
we've
had
to
account
for
it
on
the
data
operations,
team
and
the
core
platform
team
to,
and
the
data
engineering
team,
to
a
certain
extent,
there's
a
time
that
we
basically
have
to
spend
helping
less
familiar
users
understand
the
tools
that
they're
working
with
and
whether
that's
documentation.
C
You
know
meetup
type
things,
internal
meetup
type
things
or
just
pairing
with
somebody
when
they
stumble
or
run
into
a
problem.
So
it
definitely
became
a
very
different
dynamic
to
where
you
know,
rather
than
giving
you
know.
If
someone
on
the
the
data
team,
as
it
was
then
called
had
a
problem,
we
typically
expect
them
to
figure
it
out,
like
you're,
a
smart
person,
you
should
know
all
these
pieces
of
technology
that
you're
working
with
go
figure
it
out.
C
This
stack
trace
is
something
we
do
quite
a
bit
now
alex.
I
think
I
think
it
was
our
all
hands
at
the
end
of
the
year.
The
last
year
you
talked
a
little
bit
about
the
support,
change
and
moving
to
notebooks
at
the
all
hands.
Maybe
you
could
talk
a
little
bit
about
just
share
some
of
the
stuff
that
you
shared
at
the
end
of
2020.
B
A
A
B
They
are
brave
and
adventurous,
but
sometimes
it's
really
us
who
have
to
deal
with.
I
guess
the
fallout
of
this
bravery
and
but
but
it's
it's
actually
a
good
thing,
because
previously
the
fallout
would
have
been
just
so
much
worse
and
that's
why
people
weren't
as
brave
right.
So
this
is
what
it
gets
down
to
on
the
gotchas.
B
One
thing
that
we
had
to
deal
with
is
it's
so
easy
to
now
create
these
auto-scaling
clusters
right.
It's
just
a
piece
of
configuration
and
another
piece
of
configuration
is
like.
Oh
yeah
here
is
my
node
mix
up.
I
want
so
many
on-demand
nodes
versus
so
many
spots
and
then
we
so
many
times
we
get
this
request
hey.
Why
did
my
job
fail
and
you
go
look
at
it?
It's
like
well,
because
you
ran
out
of
spot
instances
or
your
spot
instances
got
decommissioned
and
somebody
else
like
well.
B
Why?
Because
you're
in
the
cloud
and
you're
competing
with
others
for
cheap
hardware?
So
that's
why?
So
so
so
I
mean,
if
you,
if
you
go
you're
in
the
cloud,
if
you're
going
to
stay
on
a
cheap
site,
you
will
you'll
have
some
situations
like
this.
Basically
is
what
I'm
getting
to.
Maybe
if
you
have
very
important
workloads,
which
will
take
two
days
to
run
whatever
that
is
right,
maybe
don't
be
so
greedy
and
get
yourself
a
little
bit
more
on
on
demand
instances.
B
D
Things
was
in
order
for
us
to
enable
that
really
advanced
bring
your
own
catalog
sort
of
thing.
We
had
to
get
into
quite
the
spiderweb
network
of
cross-account
im
policies
so
like
we
had
clusters
assuming
roles
and
other
accounts
which
had
access
to
a
third
accounts
resource
and
had
to
make
sure
that
the
resources
on
every
single
side
of
that
cross
count
barrier
had
all
the
permissions
set
up
for
both
ways
and
debugging.
That
can
be
quite
a
nightmare.
D
B
A
No
guys
this
is
actually
perfect,
not
to
knock
aws
in
the
process,
but
this
is
a
great
way
actually
to
finish
today's
session.
I
really
thank
you
all
taking
time
out
of
your
really
busy
schedules
to
participate
in
today's
session.
Really,
thank
you
very
much
for
and
for
all
of
you
who
are
watching
today's
session.
Hope
you
get
to
watch
all
three
and
get
to
learn
a
little
bit
more
from
everything.
Scribd
has
done
to
well
upgrade
themselves
from
legacy
systems
to
the
cloud
I.