►
From YouTube: Data Collab Lab: How Scribd Uses Delta Lake to Enable the World's Largest Digital Library
Description
How Scribd Uses Delta Lake to Enable the World's Largest Digital Library
Discuss with Scribd Engineers on Delta Tables and the Transaction Log
Join us for the next Data Collab Lab with Franco and Denny where we interview QP and Tyler from Scribd for a fun AMA session on How Scribd Uses Delta Lake to Enable the World's Largest Digital Library. In this session, we will discuss with Scribd engineers on how they transitioned from legacy on-premises infrastructure to AWS as well as utilize, implement, and optimize Delta tables and the Delta transaction log. Come ready with questions as this will be a fun interactive session. Databricks is proud to announce that Gartner has named us a Leader in both the 2021 Magic Quadrant for Cloud Database Management Systems and the 2021 Magic Quadrant for Data Science and Machine Learning Platforms. Download the reports here. https://databricks.com/databricks-named-leader-by-gartner
A
B
Perfect-
and
my
name
is
denny
lee-
I'm
also
a
developer
advocate
here
at
databricks
based
on
sunny
seattle.
Yes,
if
you
don't
know
which
session
you're
in
you're
in
the
house,
script
uses
delta
lake
to
enable
the
world's
largest
digital
library,
we're
gonna,
discuss
with
script
engineers
on
delta
tables
and
the
transaction
log.
Yes,
we're
gonna
get
geeky
so
make
sure
you
ask
your
questions
in
the
q
a
panel
we're
prepared
to
answer
them
now
saying
this
qp
tyler.
B
C
D
It's
really
gotten
in
deep
with
it
anyways
I'm
our
tyler
croy,
I'm
the
director
of
platform
engineering
here
at
scribd.
I
lead
the
core
platform
team,
which
builds
out
data
platform
infrastructure
and
the
data
engineering
team,
which
builds
a
lot
of
the
tooling
and
ingestion
pipelines
that
sit
on
top
of
that
platform.
A
A
This
is
all
about
you,
bringing
the
world's
toughest
problems
to
databricks
and
us
trying
to
solve
them,
live
or
talk
about
how
other
people
solve
them.
So
if
you
have
things
that
you
want
us
to
talk
about
or
to
work
through,
go
ahead
and
email
us
at
datacollab
lab
at
databricks.com
and
maybe
it'll
be
on
a
future
episode,
I
don't
know
what
do
you
think
danny.
B
Oh
absolutely
so
you
asked
some
us
some
solid
questions.
We
want
to
answer
them,
we're
going
to
answer
them
in
this
format
initially,
but
then
later
on,
we'll
probably
even
do
it
on
youtube
live.
So
that
way,
you
really
can
call
us
out
all
right
so
saying
that
let's
get
this,
this
started
tyler,
let's
start
with
you
just
because
we'd
love
for
you
to
set
the
background
first
and
tell
us
a
little
bit
about
scribd
and
why
you
have
you
know:
what's
the
world's
largest
digital
library.
D
Sure
so
scribd
in
in
startup
terms
is
actually
a
quite
an
old
company.
I
think
it
was
about
12
or
13
years
ago
now
that
scribd
started
and
scribb's
lineage
is
in
document
sharing.
A
lot
of
people
may
remember
the
era
of
startups
themes.
Script
is
one
of
them.
Flickr
and
scribd
and
tumblr
all
came
out
around
the
same
time,
and
we
were
we're
all
about
sharing
documents
on
the
internet
over
time.
A
D
Let's
call
where
we
came
from:
let's
call
that
revision
zero
revision,
zero
was
organically
created
there.
There
was
a
data
platform
that
existed
when
I
joined
scribd
that
was
largely
built
around
this
sort
of
legacy.
Hadoop
hdfs
hive
infrastructure
with
when
I
joined,
we
had
spark
1
running
as
well
as
spark
2
and
a
number
of
other
non-sparky
things
running
on
this
cluster,
and
we
just
we
like
it
was
barely
meeting
our
needs
that
we
had.
D
You
know
at
day,
one
when
I
started
much
less
future
needs
or
desired
things
that
we
wanted
to
do,
and
so
the
first
order
of
business
was
really
in
2019
evaluating.
Like
you
know,
we
can't
do
it.
We
can't
do
what
we
want
to
do
today
with
this
data
platform,
much
less
come
up
with
new
ideas
of
what
we
can
do
with
this
content.
So,
let's
find
let's
go,
find
a
way
that
we
can.
D
You
know
build
a
platform,
that's
going
to
grow
to
future
and
unconceived
of
use
cases
and
that
sort
of
set
us
down
the
road
of
you
know
looking
at
the
cloud,
obviously,
because
we
can
go
get
a
lot
of
stuff
very
easily
in
in
the
cloud
and
in
our
case
our
cloud
provider
aws.
You
may
have
heard
of
them:
they're
really
really
niche
really
niche
cloud
provider,
but
they
had
there's
a
bunch
of
things
that
were
already
built
into
aws
that
we
were
interested
in
using.
D
But
then
we
discovered
a
lot
of
what
databricks
had
been
up
to
at
the
2019
spark
nai
summit-
and
I
remember
this
very
clearly
because
it
was
in
san
francisco
and
I
didn't
go.
I
could
have
gone,
but
I
didn't
go
because
I
had
other
stuff.
I
wanted
to
do
and
multiple
engineers
came
back
from
that
session
or
from
that
event
saying
we've
got
to
do
delta
lake
delta
lake.
Like
that's
all
I
heard
about
that.
A
It's
awesome,
so
you,
when
you
ex
when
you
experimented
with
the
cloud.
Obviously
you
probably
would
have
looked
at
like
s3,
and
maybe
you
were
using.
You
know
you
mentioned
hadoop
before
you
were
on
hdfs.
When
you
first
tried
experimenting
with
s3.
Did
you
run
into
any
issues?
This
is
like
before
you've
kind
of
discovered
delta.
When
you
were
just
kind
of
dabbling
in
dipping
your
toe
in,
if
you
will
did
you
notice
any
problems
with
with
doing
that,.
D
We
fortunately
found
delta
fairly
early.
We
had
already
we
started
doing
some
prototyping
and
investigations
on
how
our
existing
workloads
would
work
on
s3
and
aws,
and
one
of
the
big
things
that
that
we
had
problems
with
is
we
don't
we
still
to
this
day?
Don't
have
a
hundred
percent
clear
understanding
of
how
all
of
the
workloads
are
actually
utilizing?
Storage
at
the
bottom
of
the
data
platform
or
building
you
know
spark
on
top
of
it
and
frankly
I
don't
want
to
know.
I
don't.
D
I
don't
really
need
to
know,
but
as
we
dug
into
some
of
these
use
cases,
we
found
people
doing
things
really
interesting
things
from
that
way
with
hdfs
or
with
hdfs
and
hive
in
concert
and
these
dependencies
between
jobs
and
we've.
We've
talked
about
this
publicly
on
our
tech
blog,
but
when
we
started
this
journey,
we
had
over
1500
tasks
that
we're
running
in
our
nightly
batch
workloads
that
we're
doing
things.
You
know
what
they're
all
doing.
D
B
All
right,
let's
switch
more
instead
of
like
this
back
and
forth,
let's,
let's
dive
into
the
technical
details.
So
let's
think
here
you,
you
talk
a
little
bit
about
the
fact
that
you
did
do
this
migration,
but
can
you
can
either
tyler
or
qp?
Tell
us
a
little
bit
more
about
like
how
did
you
make
do
that
migration?
You
had
a
lot
of
these.
It
was
it's
my
understanding
that
you
had
a
lot
of
the
these
workloads
in
hdfs
and
or
in
hive
yeah.
How
did
you
manage
that
transition.
D
C
C
For
example,
the
1500
tasks
that
tyler
mentioned
earlier
does
not
work
out
of
box
with
airflow
airflow
would
just
die
if,
when
we
tried
the
back
on
airflow,
so
we
have
to
do.
I
had
to
do
a
lot
of
optimization
on
airport
itself
and
to
make
it
work
with
the
the
scale
that
we're
at
and
I'm
very
happy
that
the
team
worked
really
hard
to
also
upstream
all
the
optimizations
that
we
did
to
the
airport
project
so
that
everyone
I'll
get
the
same
optimization.
C
We
have
implemented
in-house,
and
so
it's
a
very
lengthy
process.
We
basically
migrate
the
data
path
from
layer
by
layer,
so
the
project
started
with
migrating
the
output
layer
of
the
pipeline,
which
writes
the
data
into
external
data
stores
like
mysql
or
any
kind
of
online
data
storage
like
elasticsearch.
C
So
we
get
that
layer
migrated
and
then
we
work
on
the
middle
layer,
which
is
all
these
nested
task
dependencies
and
then,
at
the
end,
we
actually
have
planned
to
migrate,
the
upstream
layer,
the
input
data
layer,
meaning
the
the
scope
tasks
and
all
all
the
layers,
that's
responsible
for
ingesting
input
from
kafka
and
other
sources
into
the
data
link.
So
that's
how
we
plan
out
the
migration
process.
C
So
far,
I
would
say
it's
been
doing
pretty
well,
because
the
data
breaks
the
capabilities
we
get
from
databricks
elastic
cloud
and
as
well
as
really
being
able
to
spin
up
clusters
really
fast
and
being
able
to
test
all
these
codes
locally
for
each
developers
to
have
their
own
developer,
develop
workspace.
That
was
a
huge
win
for
in
terms
of
productivities
and
I've
been
really
enjoying
it.
So
far,.
A
C
D
D
So,
if
you
think
about
starting
from
the
back
of
a
big
etl
pipeline
at
the
back,
we
added
another
stage
at
the
back
of
that
which
is
right
to
delta
lag
right.
So
we're
writing
all
the
state
data
at
the
end
of
the
pipeline
into
delta
a
and
then
we
were
able
to
start
to
bring
application
workloads
over
into
you
know
aws
and
data
bricks.
On
top
of
that,
because
the
data
was
already
there,
one
of
the
other
things
that
I
briefly
alluded
to,
franco
that
we
had
all
of
these
different
types
of
workloads.
D
For
those
that
don't
know,
databricks
is
a
spark
platform.
It's
not
a
hive
platform,
it's
not
a
yarn
substitute,
it's
spark
right,
and
so
that
meant
all
of
the
wacky
workloads
that
were
like
just
hive
queries
or
doing
some
old
style.
Hadoop
map
produce
or
spark
one
like
everything
has
had
to
converge
on
spark
as
a
platform
which
is
beneficial
for
a
lot
of
reasons.
D
But
it's
also
been
a
lot
of
work,
and
when
we
started
to
dig
into
that
big
graph
of
1500
tasks,
we
found
that
almost
half
of
those
tasks
were
basically
these
templatized
hive
queries
and
hive
query.
Language
is
not
quite
the
same
as
spark
query
language,
and
we
found
some
very
interesting
abuses
of
hive
as
we're
doing
this
evaluation.
A
That's
awesome,
you
know
you
say
it
was
a
little
bit
of
a
challenge
that
you
know
that
query
runner
sounds
like
some
pretty
nifty
code,
so
I'll
be
following
along
to
see
if
you're
open
with
it,
but
there's
a
couple
of
things
there
I
kind
of
want
to
like
dig
a
little
bit
deeper
into
you
mentioned.
You
did
this
layer
by
layer
and
then
we're
talking
about
a
lot
of
spark
migrations,
and
I
kind
of
want
to
to
ask
this
means
it's
not
an
all
or
nothing
type
of
migration
effort.
A
D
As
qp
mentioned
like
this
is
definitely
an
ongoing
project
as
we
like
literally
today,
right
now,
our
nightly
batch
processes
are
on
both
our
legacy
data
infrastructure
and
on
data
bricks
from
being
driven
from
the
same,
an
airflow
dag.
I
actually
qp.
Do
you
happen
to
know
like
what?
How
far
along
like
what
percentage
or
some.
A
D
C
We
started
there's
one
thing
that
I
would
like
to
point
out
that
why
we
went
with
this
approach
instead
of
like
a
fork
and
lift
approach,
and
we
actually
thought
about
that.
A
lot
at
the
beginning
and
the
reason
we
did
with
this
layer
by
layer
approach
is
we
want
to
deliver
incremental
deliverables
to
our
customers,
internal
customers,
for
example.
The
data
sync
process
that
tyler
mentioned
earlier,
which
synchronized
the
hdfs
data
into
s3
delta
leak.
C
We
were
able
to
reduce
latency
for
our
apis
and
and
basically
get
got
rid
of
the
latency
spike
during
the
export
process.
So
that
was
the
main
reason
that
we
went
with.
This
approach
was
really
to
give
incremental
data
deliverables
to
the
engineering
team,
and
instead
of
you
know,
let's
wait
for
a
couple
months
or
a
year
before
we
can.
Everyone
can
enjoy
all
the
benefits
that
we
get
from
the
cloud
and
databricks.
B
Cool,
so,
oh
sorry
about
that,
I
forgot
that
I
muted
myself
and
was
busy
talking
and
forgot
to
do
that.
So
I
this
sounds
amazing.
What
you're
talking
about
qp?
Well,
the
what
so,
what
kinds
of
things
that
the
consumer's
been
saying
about?
You
know
basically
using
delta
and
databricks.
You
know:
has
their
experience
itself
been
better
by
using
the
platform
now
that
you've
started
switching
over
as
you've
been
sort
of
calling
out.
C
Yeah,
I
think
the
at
least
from
my
point
of
view.
The
two
biggest
thing
that
I've
heard
from
other
developers
are
the
first
one
is
being
able
to
work
on
notebooks
in
a
collaborative
way
like
being
able
to
have
a
look
that
you
can
share
with
other
people.
Actually,
everyone
is
jumping
in
the
same
ui
and
collaborate
on
that.
That
was
a
huge
win.
C
It's
really
easy
to
share
results
like
you
can
easily
build
ad
hoc
that
dashboards
that
way
as
well
and
share
with
the
rest
of
engineering
like
with
a
couple
clicks.
So
that
was
a
big
win
and
then
also
we
before
doing
that,
we
were
using
important
for
ad
hoc
queries
as
well,
and
that
was
a
lot
faster
than
hype.
We
we
were
concerned
about.
You
know
doing
that.
The
same
queries
on
data
databricks,
notebooks
and
thinking
that
would
be
slower
with
spark.
C
We
read
some
blog
posts
from
databricks
claiming
that
it's
a
lot
faster
than
impala,
so
we
we
weren't
really
believing
that,
but
actually
trying
that
ourselves
on
databricks.
It
did
turn
out
that
the
same
query
actually
even
ran
faster
on
databricks
than
in
partner.
So
that
was
also
a
huge
win
for
us
as
well,
and
then
the
other
common
thing
that
I've
observed
is
in
the
past
when
we
were
working
with
legacy
on-prem
data
infrastructure,
it's
a
lot
harder
to
get
new
boxes
right.
C
We
basically
don't
have
to
have
to
fire
tickets
to
get
a
new
machine
for
development
purpose,
so
people
there
were
times
where
people
were
actually
contending
for
hey.
Can
you
can
you
give
your
developer
developer
machine
to
me
so
that
I
can
run
my
code
like
that
kind
of
discussion
with
databricks
and
the
cloud?
We
don't
have
that
problem
anymore,
like
anyone,
anyone
can
just
spin
up
as
new
machines.
They
want
and
develop
on
the
new
work.
So
that
was
the
hood.
B
D
B
You're
you're
this
you're
pulling
back
old
memories
on
this
one.
Yes,
where
you're
pulling
off
old
hive
queries
like
this
is
a
few
years
back
now
for
me
as
well,
but
same
idea,
right,
I'd,
go
ahead
and
right
before
lunch
or
more
usually
right
before
I
left
work
right,
because
I
knew
it'd
run
overnight
right.
So
just
before
I
end
I
just
shoved
to
the
cluster,
just
shoved
a
whole
bunch
of
queries,
let
it
run
and
then
hope,
and
then
maybe
I
would
log
in
like
at
midnight
just
to
check.
So
he
was
okay.
D
B
D
That's
the
biggest
problem
with
hive
is
that
every
user's
query
is
going
to
try
to
command
the
resources
of
the
cluster
and
now
that
we're
having
users
that
are,
you
know,
spinning
off
clusters
for
their
own,
their
own
work,
or
you
know
when
we
launch
jobs,
we're
launching
clusters
per
job.
That
means
we
have
a
much
more
efficient
compartmentalization
of
those
resources.
A
That's
awesome:
I'd
like
to
pivot
to
another,
very
common
problem
that
I
that
I
used
to
experience
and
probably
maybe
other
people
have
as
well.
You
know
that
the
catalog
the
metadata
catalog,
so
you
went
from
the
hive
metastore
to
glue.
Can
you
tell
us
a
little
about
that?
How
you
managed
it
how
it
worked
out?
You
know
how
things
are
are
working
now
with
that
you
integrated
delta.
D
So
we
we
adopted
aws
glue,
catalog
and
it's
gone
okay,
it's
not
my
favorite
part
of
the
stack,
but
I
don't
know
anybody
who
gets
super
excited.
I
mean
maybe
there's
people
at
databricks.
I
get
super
excited
about
catalogs,
I'm
definitely
not
a
person
that
gets
super
excited
about
the
the
big
challenge
that
we
had
with
the
migration
to
glue
catalog
is
that
we
had
to
maintain
two
catalogs.
Basically,
because
we're
still
you
know,
we've
got
one
foot
in
our
legacy
data
center
and
then
one
foot
in
aws.
D
So
we
spent
a
fair
bit
of
time
writing
that
to
make
sure
that,
from
a
developer
standpoint
that
we
weren't
spending
a
lot
of
time,
trying
to
manually
manage
multiple
catalogs,
that
we
just
have
this
one
single
source
of
truth.
That's
in
a
git
repository
that
defines
here's
what
the
schema
looks
like
for
all
of
our
catalogs,
the
other.
There
is
a
slight
nuance
that
we
didn't
understand
about
glue
catalog
when
we
started
to
use
it
a
glue.
Catalog
is
one
per
aws
account.
D
So,
if
you
have
you
know,
if
all
of
engineering,
for
example,
is
using
one
aws
account
for
their
their
their
work,
then
you
have
one
glue
catalog
for
all
of
those
teams.
In
our
case,
we
actually
use
multiple
aws
accounts
for
different
teams
to
give
people
a
little
bit
more
of
a
sandbox.
So
they
they
don't
conflict
with
anybody
else's
work,
but
that
sort
of
accelerated
the
need
for
automated
tooling
to
manage
these
catalogs.
D
Because
all
of
a
sudden
we've
got
a
catalog,
that's
associated
with
the
applied
research
team
and
a
catalog
for
the
recommendations
team
in
their
aws
account
in
a
lot
of
cases,
they're
pointing
to
the
same
data
underneath
in
delta
lake,
but
they're
in
these
different
accounts
in
aws.
And
so
we
have
to
you
know,
push
changes
to
all
of
these
different
accounts
which
can
be
challenging.
If
you
don't
have
the
automation
already.
B
Interesting,
so,
okay,
let
me
switch
off
the
catalog
topic,
since
you
called
that
out,
tyler
and
qp.
I
want
to
know
a
little
bit,
and
this
is
actually
some
of
the
questions
that
I've
been
coming
in
from
both
linkedin
live
as
well
as
youtube
about.
How
do
you
optimize
the
match,
the
file
sizes
in
your
cloud
right,
for
example,
when
you
have
a
lots
of
files,
especially
going
to
your
s3
buckets
right
that
potentially
can
increase
the
cost
right.
So
how
do
you
optimize
all
this?
B
C
B
Perfect.
Well
then,
oh
sorry,
I
just
wanted
to
do
a
quick
follow-up.
If
that's
okay,
we're
related
to
this,
then,
is
talk
a
little
bit
about
the
streaming
and
batch
concept
right.
The
fact
that
you
actually
need
to
run
those
optimized
jobs
to
handle
the
fact
that
you're
streaming
and
handling
both
streaming
and
batch
concurrently
at
the
same
time
right
because
that
ultimately
increases
the
number
of
files
that
you
actually
are
writing
into.
S3
right
you
when
you're
streaming,
there's
all
these
small
little
files
that
you
had
alluded
to
earlier
so
yeah.
C
So
for
from
the
writers
product
right,
they
don't
really
care
about
optimization
like
they
just
write,
whatever
data
they
want
to
the
table,
and
you
can
do
concurrent
writes
as
well
and
on
the
side.
We
also
have
a
dedicated
like
optimized
job,
that's
occupated
by
airflow
that
runs
on
a
specific
cadence
and
this
job.
C
We
actually
have
a
configuration
for
to
configure
like
how
frequent
do
we
want
to
optimize
for
a
specific
table
in
in
delta
lake
and
that
job
is
responsible
to
you
know
optimize
all
these
small
file
problems
and
in
the
background
for
the
other
writers
and
so
the
writers.
Basically,
they
don't
have
to
care
about
the
small
file
problems.
C
D
Man,
I
love
a
good
manager
question.
Yes,
let's
talk
about
value
so
when
I
drink
when
I
joined
scribd
like
streaming
or
real-time,
data
was
probably
that
that
was
like
that
pie
in
the
sky
sort
of
moonshot
initiative
that
that
we
had
started
to
work
on
the
way
that
most
of
our
data
customers
traditionally
have
in
consumed
data.
D
The
the
big
one
was
totally
unexpected
was
around
our
ad-hoc
queries,
like
we
enabled
all
of
these
people
to
use
databricks
notebooks
to
run
these
add-on
queries,
but
because,
as
as
qp
mentioned,
because
we're
like
streaming
to
a
delta
table
from
the
user's
perspective,
it
just
looks
like
a
table
like
it.
Does
it's
not
this
special
thing?
D
It's
you
know
if
you,
if
you
want
to
pull
streaming
data
into
your
your
ad
hoc
workloads
and
you
don't
have
delta,
you
might
be
teaching
users
how
to
connect
to
kafka
topics
or
pulling
it
into
some
other
intermediate
store
that
they're
going
to
query.
But
for
our
users
a
delta
table
that
stream
two
versus
a
delta
table,
that's
updated
via
the
nightly
patch.
D
Once
we
get
started
to
add
some
streaming
data
into
the
platform
they're
over
the
moon
excited,
I
think,
the
the
fastest
time
from
data
generated
to
something
being
available
in
the
platform
that
I've
seen
is
about
nine
seconds
and
that's
like
nine
seconds,
from
the
event
being
created
from
a
production
web
application
to
it
being
available
for
a
databricks
novel
book,
and
when
you
take,
when
you
take
that
value
proposition
to
somebody
who's
used
to
having,
like
worst
case
48
hours
for
their
data
they're
like
wait.
What
what
do
you
mean
nine
seconds?
D
That's
not
like
it's!
It's
like.
If
you
showed
a
spaceship
to
someone
from
the
1700s
like
they
almost
can't
even
comprehend
the
tremendous
amount
of
change
that
they
just
encountered
and
get
to
benefit
from
so,
unfortunately,
not
all
of
our
data
is
streaming.
We,
we
do
have
a
lot
of
data
that
streams
through
kafka
into
different
delta
tables,
but
there
are
some
things
like
qp
mentioned:
scoops
are
at
the
top
of
our
ingestion
pipeline.
That's
I
mean
scoops,
that's
with
sqop,
if
anyone's
not
familiar
with
it.
D
It's
just
copying
my
sql
data
into
into
the
data
platform.
That's
not
streaming
and
that's
still
done
on
a
daily
batch
and
getting
that
data
to
be
streaming
is
actually
really
really
hard.
So
we
might
not
do
it,
but
for
the
day
that
can
be
streamed,
it's
pretty
sweet
once
it's
streamed
into
the
into
delta
tables.
B
I'm
just
curious:
are
you
by
just
using
things
like
trigger
once,
or
things
like
that,
so
you
can
convert
existing
batch
queries
into
much
more
much
more
like
near
real
time.
So
that
way,
you
don't
actually
have
to
maintain
all
these
different
jobs.
Admittedly
enough,
I
don't
think
I've
ever
tried
trying
to
do
that
with
a
scoop
query
immediately
enough.
C
So
the
10,
the
salvador,
we
were
actually
basically
doing
the
migration,
but
that's
some
of
the
optimizations
that
we're
looking
forward
to
do
in
the
next
year.
Like
really
up
there's,
we
can
see
a
lot
of
low-hanging
fruit
for
our
data
pipeline
and
I
can
see
that
a
lot
of
the
batch
jobs
can
be
optimized
into
like
streaming
jobs
or
incremental
processing
jobs.
That,
I
think,
will
give
us
like
an
order
of
magnitude
improvement
to
our
own
time
as
well
and
including
the
data
freshness
at
the
end
of
the
user.
B
B
B
A
A
C
So
I
I
don't
think
we
have
done
like
a
comprehensive
evaluation
of
that,
but
at
least
from
my
understanding,
the
most
obvious
effect
that
we
had
us
doing
at
home.
Queries
like
enabling
the
cache
was
not
enabling
the
cache
made
a
big
difference
when
you're
doing
ad
hoc
queries
on
the
data
set.
We
have
so.
We
definitely
leverage
that
a
lot
in
our
notebooks
ad
hoc
clusters,
where
users
get
a
lot
faster
queries
through
the
caching.
D
D
One
of
these
ad
hoc
commands
spark
is
going
and
grabbing
those
those
data
that
data
off
of
s3
and
then
pulling
it
into
the
cluster
to
process
it
right
as
I
hunt
and
peck,
and
try
to
get
the
spark
sql
query
to
do
the
right
thing,
each
indication,
if
I'm
just
grabbing
that
data
off
the
local
disk.
It's
definitely
faster
part
of
the
reason
why
this
is
a
little
bit
difficult
for
us
to
quantify
like
we're
relatively
new.
I
say
you
know
less
than
six
months.
D
No,
maybe
it's
about
six
months
now,
because
it
is
october
already
about
six
months
that
we've
been
running
a
lot
of
our
ad-hoc
workloads
and
increasing
amounts
of
our
automated
workloads
in
in
data
breaks
on
delta
lake
there's.
So
many
low
hanging
pieces
of
fruit
that
oftentimes,
where
we
see
big
performance
gains,
are
like
these
boneheaded
mistakes
that
we've
made
along
the
way.
Like
a
user.
D
I
use
one
of
my
favorite
examples
was
I
remember
this
one
job
where
the
user
was
inadvertently,
pulling
an
entire
result
set,
which
was
like
gigabytes
of
data
and
then
squeezing
it
through
a
udf
and
just
not
trying
to
pull
the
entire
result
set
from
a
single
partition
and
squeezing
it,
through
a
udf,
made
a
huge
impact
on
our
performance
and
huge
impact
for
their
queries.
And
it's
just
like
these
very
simple,
like
use
the
tool,
maybe
a
little
bit
better.
D
Things
have
made
the
biggest
dramatic
impacts
on
our
on
our
performance,
because
it
is
just
a
checkbox
and
for
anybody
that's
not
used
databricks.
When
you
launch
a
cluster,
it
really
is
like
check
delta
cache.
We
haven't
needed
or
desired
to
do
a
really
substantial
back
off
of
like
using
delta
caching
versus
not.
We
just
turn
it
on
said.
D
Well,
you
know,
even
even
like
a
20
cash
hit
rate
is
still
better
than
zero
percent
and
it
doesn't
cost
us
anything
to
turn
it
on
and
it's
a
check
box,
so
we'll
just
check
the
checkbox
done,
and
so
some
of
those
performance
things
where
it's
like.
Oh,
if
we
just
check
this
box,
we
might
get
better
performance.
We
just
check
the
box
and
you
know
assume
that
somebody
has
done
the
actual
bake
off.
You
know
on
the
database
side
or
somewhere
else,
and
if
we
get
the
performance
benefit
from
it,.
A
B
So
let
me
switch
a
little
bit
more
to
the
the
process
around
engineering
right
so
outside
of
just
clicking
on
checkboxes.
That
is
okay,
I'm
just
curious.
How
and
maybe
I'll
direct
this
to
uqp
how's
databricks
help
from
with
your
engineering
team,
the
ability
delivery,
don't
worry
tyler!
I
still
want
you
to
chime
in
I
just
figured.
I
want
qp
to
talk.
You
know
from
the
instead
of
the
manager
talking
first,
I
want
the
geek
to
talk
first,
that
was
all.
C
I
think,
although
not
all
the
optimizations
of
open
source,
but
at
least
when
we're
using
the
database
platform,
we
get
a
lot
of
the
optimizations
that
we
we
needed
to
get
the
job
done.
A
lot
faster
and,
like
I
said
earlier,
like
in
the
old
days,
engineers
have
to
compete
for
develop
development
machines.
That
was
not
the
case
anymore,
be
able
to
collaborate
on
the
notebooks
that
was
huge
win
and
then.
C
And
then
the
other
thing
is
less
about
database,
but
more
about
running
your
development
workforce
in
the
cloud
where
you
can
actually
scale
to
any
kind
of
any
machines.
You
want
to
get
your
work
done
like
if
you
you,
if
you
need
this
one
to
get
done
faster,
you
just
add
more
machines
and
it
would
get
them
faster
and
so
that
that
was
a
big
win
for
us
as
well
yeah.
B
C
Yeah,
so
we
we
are
actually
working
with
databricks
on
getting
ml
flow
into
production
before
our
production
use.
There
are
some
limitations,
but
we're
actually
working
closely
with
the
team
and
I'm
very
looking
forward
to
the
upcoming
improvements
from
the
team
and
right
now.
We
actually
are
storing
our
models
in
mflos
and
we're
also
using
it
to
do
model
tracking
as
well,
for
machinery
models.
A
A
You
know
things
I'm
home
a
lot
more
and
I
have
more
time
to
read
to
consume.
The
written
word
either,
and
we
talked
about
you
know
is-
is
consuming
an
audio
book.
Do
you
call
that
reading?
I
call
it
reading.
I
actually
was
talking
to
an
old
friend
over
the
weekend
and
I
was
like
oh
so
I
was
reading
this
audiobook
and
was
like
what
are
you
talking
about.
A
I
was
like
listen
man
just
just
go
with
it,
but
anyway,
so
I've
been
consuming
a
lot
more
media
and
I've
been
a
spotify
user.
Since
I
don't
know
the
early
2010s
when
it
first
came
out
and
at
this
point
spotify
spotify
has
had
my
likes
and
my
data
now
for
10
years
and
their
algorithms
they're
really
good
at
knowing
what
I
like
and
they
kind
of
queue
up
these
automated
playlists.
A
That
kind
of
give
they
find
all
the
artists
I
like,
and
they
even
find
new
music
for
me,
and
I
love
it,
I
absolutely
I
don't
even
listen
to
the
radio
anymore.
Spotify
does
a
much
better
job
of
finding
the
content
I
like
I
would
like
and
delivering
it
to
me
is
scribd
doing
anything
like
this
for
consuming
the
written
word
and,
and
can
you
tell
us
a
little
bit
about
what
you
might
be
working
on.
D
I
can
I
can
talk
a
little
bit
about
the
I'd,
say
the
more
product
and
business
side,
and
then
qp
can
talk
about
some
of
the
technology.
That's
underneath
or
that
we're
working
on
any
recommendations
is
probably
one
of
the
most
important
parts
of
our
future.
I
would
say,
like
the
what
originally
really
attracted
me
to
scribd
as
a
company
is
like
the
business
relies
on
the
data
platform.
The
future
success
of
scribd
is
really
really
intertwined
with
how
well
we
can
build
out
and
scale
and
mature.
D
You
know
in
you
know,
like
the
classic,
like
bank
type
of
infrastructure,
there's
a
very
specific
purpose
of
what
the
data
engineering
team
is
there
to
do,
and
and
what
they're
generating
and
what
their
output
does.
It's
not
really
going
to
change
the
trajectory
of
the
bank
right
with
scribd,
because
data
is
so
core
and
you
know
the
content,
the
the
books,
the
audiobooks
and
the
documents
is
so
core
to
what
makes
script
valuable
and
what
makes
scripts
successful.
D
If
we
make
a
better
data
platform,
if
I
can
enable
that
recommendations
engineer
to
do
a
better,
whatever
the
hell
they
do,
then
then
that's,
you
know
immediately
more
success
for
the
company
and
so
for
us
or
for
me
at
least
the
the
fact
that
you
know,
recommendations
and
search
are
so
crucial
to
the
business
and
that
our
work
on
the
data
platform
directly
impacts
that
very,
very
key
functionality
is
really
really
exciting,
but
it
also
means
that
we've
gotta
do
things
right.
No,
no
screwing
around
here,
hey
sqp,
you're
gonna,
say.
C
Just
to
cycle
about
a
little
bit
to
the
technical
side
of
things.
So
since
this
is
a
this-
is
a
delta
tech
link
talk.
I
want
to
mention
how
teleduct
enables
to
build
better
up
recommendation
systems.
So,
as
tyler
mentioned
earlier,
we
have
this
nightly
batch
pipeline
thread
that
runs
that
read
every
day
and
you.
A
C
User
requests
in
real
time
or
near
real
time
and
then
provide
much
better
and
much
fresher
recommendation
to
our
users.
And
I
think
that
itself,
it's
a
proof
that
you
know
having
the
right
technology
will
unlock
all
these
possibilities
to
the
engineering
teams
and
from
the
teams
to
build
products
that
was
not
even
possible
before.
So.
I
think
that
was
a
big
thing
we
could
get
from
using
the
other
rig
as
well.
A
That's
awesome
I
just
kind
of
want
to
go.
I
want
to
kind
of
synthesize
a
couple
of
things
that
we've
been
talking
about
here,
so
you're
telling
me
that
you're
starting
to
leverage
more
streaming
technologies.
That
means
that
your
data
is
getting
refreshed
faster,
which
means
the
the
the
ability
for
your
analysts
to
query
more
up-to-date
data
and
analysts
and
scientists.
I
need
any
data
worker
right,
so
their
ability
to
get
more
fresh
data
has
been
increased,
but
not
only
that,
but
with
sql
and
with
all
the
programming
languages
that
we
offer
on
the
platform.
A
So
you
could
use
python
scala,
sql
r,
but
now
their
ability
to
consume
that
data,
because
the
databricks
runtime
is
more
efficient.
We
have
things
like
delta,
caching
that
they're
not
only
able
to
get
the
data
latency,
so
the
refresh
rate
of
the
data
but
they're
actually
able
to
query
the
data
faster.
So
their
ability
to
to
derive
insight
and
take
action
is
significantly
improved.
With
this
platform
is.
C
A
B
B
We
got
a
perfect
one
for
you
here,
okay,
one
of
the
questions
that
actually
had
been
off
the
side,
but
I
figured
you'd
actually
do
a
great
job
explaining
it
is
that
to
get
delta
transactions
right.
Do
you
not
need
to
copy
the
data
it
was?
It
was
the
the
question
was
like
an
understanding
from
one
of
my
previous
sessions.
I
was
explaining
that
in
some
cases
you
need
to
copy
the
data.
In
some
cases
you
don't
like,
for
example,
in
place
conversions.
B
C
C
So
it
depends
on
the
usage
pattern,
so,
if
you're
doing
append
only
like,
if
you're
only
adding
new
data,
then
there
is
no
no
copy
involved.
So
what
delta
degree
does
from
my
point
of
view?
Is
it
provides
it
under
the
hood?
Actually,
the
main
thing
it
does
is
to
look
up
like
what's
the
list
of
product
files
that
you
should
read
for
for
the
current,
for
the
time
that
when
you
requested
the
table
so
so
that
you
can
get
a
consistent
view
of
what
the
data
is
in
the
table
right.
C
So
what
it
does
really
is
to
resolve
the
list
of
product
files
that
that
the
spark
application
have
to
read
and
if
you're
only
doing
append
only
then
you're
just
adding
new
files
to
the
list.
All
right
and
that's
fine
and
copy
files
happens
when
you
have
to
do
compaction.
So
like
says
we
discussed
earlier
when
you
have
to
merge
small
files
into
one.
C
That's
when
you
have
to
copy
those
small
files
into
one
big
file
right,
that's
when
copy
power
happens
and
all
it
happens
when
you're
doing
up
search,
meaning
where
you
want
to
update
a
you
know
couple
rows
in
a
single
parque
file.
Then
you
have
to
you,
know,
copy
the
whole
project
file
and
then
update
with
the
updated
version
and
write
the
new
version
and
and
then
basically
in
the
transaction
log,
it
manage
the
the
history
so
that
it
knows
that
the
old
product
file,
we
don't
have
to
read
them
anymore.
C
B
Rock
on
well,
then,
let's
go
ahead
and
for
our
final
question,
just
because
I
want
two
minutes
left,
so
we
can
do
the
final
little
close
down
for
franco,
myself.
You've
talked
about
a
lot
about
the
pins.
Well,
let's
talk
about
impen
versus
merge
when
it
comes
to
current
concurrent
writers
or
readers
on
a
single
delta
table.
C
So
when
you're
doing
concurrent
rights
from
with
append
only
it's
totally
fine
like
there's,
you
don't
get
conflicts
at
all.
So,
under
the
hood,
the
data
is
using
optimistic,
concurrency
control,
meaning
that
all
the
writers
try
to
write
to
a
table
as
if
there
is
no
conflict
and
when,
when
there
is
a
conflict,
then
they
try
to
resolve
the
conflict.
So
when
you're
doing
a
pen
only
there
is
not
going
to
be
any
conflicts
like
if,
when
they
retry,
they
will
just
add
the
new
file
to
this.
C
So
when
you're
doing
up
search,
there
could
be
conflict
depending
on
the
application
logic.
So
in
some
cases
you
might
actually
get
an
error
that
you
you
have
to
handle
at
that
application
level,
but
if
sometimes
if
the
conflict
can
be
resolved
delta,
like
will
also
resolve
the
conflict
automatically
for
you
as
well.
B
Perfect,
oh
thanks
very
much
qp.
I
apologize
in
advance
for
everybody
who
had
asked
questions,
but
we
weren't
able
to
answer
them.
We
are
still
trying
to
balance
all
the
three
different
platforms
right
now
and
resolve
that
be
before
you
guys
go
since
we've
got
three
minutes
left.
I
did
want
to
go
ahead
and
do
a
quick
call
out
before
franco
and
I
thank
everybody
for
attending
that
is,
but
the
quick
call
out
here
actually
is
on
next.
B
Excuse
me
next
tuesday,
there's
a
the
next
session
that
we're
gonna
have
here
on
the
data
and
ai
online
meetup
will
be
a
tech
chat
from
bi
to
lake
house.
Part
two:
electric
boogaloo!
No,
there's
no
reason
why
we
actually
call
that
out.
We
just
figured.
We
would
go
do
that.
But
yes,
it's
a
session
between
myself
and
simon
whitley.
We
are
a
bunch
of
microsoft,
sql
server
geeks
that
have
now
wholeheartedly
been
in
the
realm
of
lake
houses.
So
we
would
love
you
to
come
to
that
session.
B
You
can
definitely
ask
questions
and
this
time
we
might
actually
answer
them
because
we
middle
enough
simon
and
I
had
too
much
fun.
The
last
time,
so
we
were
busy
telling
stories
and
forgot
to
actually
answer
some
of
your
questions.
We
apologize
for
that
so
tyler.
I
also
know
that
you
want
to
definitely
do
a
strong
call
out
to
the
script
technical
engineering
blog
buying
chess.
Can
you
qp
or
tyler?
Can
you
go
ahead
and
post
it
on
and
then
talk
and
do
your
shout
out
please.
D
I
I
can
talk
about.
I
don't
know
where
you
want
me
to
put
it,
but
just
at
tech.script.com,
we've
actually
written
a
lot
about
this
process,
including
some
more
of
the
details
of
the
I'll,
say
the
benefits
and
challenges
we've
had
as
we've
adopted
delta
lake.
If
you
just
go
check
it
out,
qp's
got
a
lot
of
really
good
content
that
he's
posted
as
well
around
our
migration
to
airflow
and
some
of
the
work
that
we've
done
there.
A
A
I
always
want
to
call
it
out
again
if
you
have
tough
problems,
you're
trying
to
solve,
and
you
need
some
help-
data
collab
lab
at
databricks.com
email,
us,
your
questions
or
your
your
issues
and
we'll
collaborate
together,
maybe
an
episode
where
we
invite
you
on
and
we'll
figure
it
out
together.
Thank
you,
qp.
Anyone
else
want
to
do
an
an
exit.