►
From YouTube: Delta Lake 0.7.0 + Spark 3.0 AMA
Description
On August 18th, 2020, join Apache Spark and Delta Lake committers Burak Yavuz, Tathagata Das, and Denny Lee for an illuminating “Ask Me Anything” session. Whether you would like to know more about the history of Apache Spark to the current bleeding edge use cases of Spark 3.0 and Delta Lake, this is the session to ask your questions!
Learn more at delta.io
---
Speakers:
Burak Yavuz is a Software Engineer and Apache Spark committer at Databricks. He has been developing Structured Streaming and Delta Lake to simplify the lives of Data Engineers. Burak received his BS in Mechanical Engineering at Bogazici University, Istanbul, and his MS in Management Science & Engineering at Stanford.
Tathagata Das is a Staff Software Engineer at Databricks, an Apache Spark committer and a member of the Apache Spark Project Management Committee (PMC). He is one of the original developers of Apache Spark, the lead developer of Spark Streaming (DStreams) and is currently one of the core developers of Structured Streaming and Delta Lake. Previously, he was a grad student in UC Berkeley at AMPLab, where he conducted research about data-center frameworks and networks with Scott Shenker and Ion Stoica.
Denny Lee is a developer advocate at Databricks, where he works on Delta Lake, Apache Spark, Data Sciences, and Healthcare Life Sciences. He has previously built enterprise DW/BI and big data systems at Microsoft including Azure Cosmos DB, Project Isotope (HDInsight), and SQL Server as well as the Senior Director of Data Sciences Engineering at SAP Concur. Denny holds a Masters in Biomedical Informatics from Oregon Health Sciences University. Databricks is proud to announce that Gartner has named us a Leader in both the 2021 Magic Quadrant for Cloud Database Management Systems and the 2021 Magic Quadrant for Data Science and Machine Learning Platforms. Download the reports here. https://databricks.com/databricks-named-leader-by-gartner
A
Have
pushed
the
youtube
stream
live,
welcome
again
everyone
we
have
denny,
barack
and
td
here
to
talk
about
delta
lake
0.7
and
spark
3.0
if
you
haven't
subscribed
to
the
meetup.
If
you
just
found
this
link
from
one
of
your
friends
or
colleagues,
please
do
subscribe
to
the
meetup
to
hear
more
about
our
upcoming
sessions.
Denny
has
been
organizing,
I
think
two
a
week
here,
so
lots
of
great
content,
including
one
I
think
on
thursday
right
denny
is
that
one
on
thursday
coming
up
about
glow?
A
B
B
Perfect,
well
thanks
very
much
everybody
thanks
ryan
for
introducing
everybody,
who's
on
zoom,
webinar
or
youtube,
live
gr,
grad,
yeah
glad
to
have
you.
This
is
just
in
case
you're
wondering
what
you
stumbled
in
on
this.
Is
the
delta
lake
zero
dot
sum
not
zero,
plus
spark
three
dot
ama
with
baroque
td
and
myself
all
right
just
as
ryan.
B
It
called
out
if
you
love
this
type
of
material,
please
log
in
to
our
databricks
youtube
channel
to
sign
up
and,
as
we
noted,
the
recordings
and
the
slides
are
gonna
be
available
after
this
tech
talk
or
webinar
note
that
it's
on
youtube
live
anyway,
so
basically
it'll
just
be
there.
Everyone
is
muted.
B
So
please
put
your
questions
in
the
q
a
panel
okay,
and
then
that
way
we
know
what
questions
you're
going
to
be
asking
one
of
the
things
we'll
note
is
that
we're
actually
going
to
present
some
slides
to
encourage
folks
to
basically
ask
some
questions.
Okay,
so
right
now
we're
going
to
start
off
explaining
some
of
the
features
that
are
available
in
delta
lake
0.7.0
all
right.
B
So
the
quick
call
out
here
is
that
delta
lake
0.700
is
the
first
release
on
apache
spark
3.0
and
adds
support
for
metastore
defined
tables
and
sql
ddls,
so
pretty
cool.
Now
that
we're
on
spark
3.0
woohoo,
yes,
all
right
so
now.
What
do
we
mean
by
this
support?
Okay,
so
some
examples.
Okay-
and
this
is
the
part
where
we're
gonna
say
hey
if
you've
got
questions,
please
prop
them
in
the
q
a
panel
because
we'd
love
to
actually
hear
them.
B
Okay
for
td
and
barack
they're
probably
going
to
be
doing
most
of
the
answers
so
brock.
Two,
since
right
now
I'm
presenting,
I
actually
can't
see
the
q
a
panel
because
the
way
zoom
works.
So
if
you
see
any
cool
questions
or
you
want
to
add
in
some
color
commentary
as
we
go
through
these
slides
by
all
means-
let's
go
do
that
here:
okay,
all
right!
So,
let's
start
off
with
the
context
here:
support
for
dividing
tables
in
the
hive,
metastore
right.
B
B
There
we
go
and
use
the
table
name
in
all
sql
operations,
and
so
what
do
we
mean
by
that
there
are
things
like
we
can
create
the
table.
For
example,
your
standard
create
table.
Like
events,
you
have
your
schema,
you
specify
using
delta.
So
in
other
words,
you
can
write
this
as
sql
operations,
okay,
as
opposed
to
necessarily
just
using
the
scala
or
the
python
api,
but
just
as
important
all
right.
B
You
can
also
use
the
alter
table
statements,
so
you
can
actually
add
your
columns
as
well
so
making
changes
to
the
table
and
or
the
schema
in
the
process.
So
it's
pretty
cool
that
you're.
Actually,
you
have
the
capability
of
doing
that
now:
cool
td,
barack
any
any
color
commentary
or
questions
that
you
can
probably
answer
right
now.
C
The
one
comment
that
I
want
to
add
is
like
this.
These
comment
commands
looks
like
they
were
working
in
previous.
You
know
releases
of
delta,
lake
and
spark,
but
the
thing
was
you
know:
all
of
this
information
was
actually
stored
in
the
meta
store
they
weren't
actually
delegated
to
delta,
in
the
correct
way
that
it
should
have
been.
Therefore,
now
what
we
mean
by
you
know,
there
is
support
for
things
in
the
hive.
B
Cool
so
friendly
reminder:
if
you
have
questions,
please
put
them
in
the
q
a
panel
as
opposed
to
the
chat
panel,
but
actually
there's
a
great
question
in
the
chat
panel.
So
I
figured
I'll
ask
either
td
barack
to
answer
this
question
right
from
the
get-go
is.
Can
you
help
us
understand
more
about
the
metastore.
C
Yeah
sure
I
mean
so
what
is
the
metastore?
The
metastore
is
a
let's
say,
a
construct
that
you
know
currently
is
backed
by
hive,
which
we
call
the
hive
metastore
and
what
it
does
is
that
it
stores
all
the
metadata
about
all
your
tables
in
it.
So
what
does
it
do?
It
can
store
like
the
schema
of
your
tables.
It
can
store
where
the
table
is
saved,
on
some
object,
store
or
some
jdbc
uri,
or
you
know
whether
it
can
even
be
kafka.
For
example,
it
can
save
like
a
kafka,
uri
and
whatnot.
C
So
what
it
does
is
that
you
know
like
it's,
basically
a
database
that
stores
all
this
information.
It
can
store
like
partitioning
information
for
like
tables
which
are
backed
by
parquet,
for
example,
and
you
know
whenever
you
query
your
tables.
What
spark
does
is
that
you
know
this
identifier,
for
example.
In
this
case
events
we
will
go
to
the
meta
store
to
figure
out.
Does
this
table
really
exist?
C
What's
its
schema?
Where
is
it
located?
Which
format
do?
I
have
to
use
to
read
this
table
and
query
this
table
and
all
that
information
is
gathered
through
the
metastore,
typically
with
delta
tables.
What
happens
is
there's
just
a
table
name
the
fact
that
it's
stored
as
delta
in
the
meta
store
and
the
location
of
where
it's
stored
at
so
it's
kind
of.
Like
a
you,
know,
layer
of
indirection,
for
you
know
you
not
having
to
remember
long
paths
about
you
know.
Oh,
is
this
stored
on
hdfs?
D
D
Ahead,
please
add
a
bit
more
color,
I
think.
Well,
first
of
all,
barack
is,
I
think,
selling
himself
short
he's,
not
saying
the
fact
that
he
has
spent
last
two
years
working
with
the
apache
spark
community
to
add
this
support.
So
basically,
before
spark
3.0
spark
did
not
expose
the
right
set
of
apis
to
customize.
D
This
spark
to
metastore
interaction,
so
spark
used
to
always
read
the
schema
and
other
associated
metadata
from
the
meta
store,
given
a
table
name
that
was
not
customizable
in
delta.
We
wanted
exactly
that.
We
wanted
to
customize
it
so
that,
instead
of
reading
it
from
the
meta
store,
it
reads
it
from
the
transaction.
Log
and
burak
did
all
the
effort.
C
C
B
Perfect
thanks
very
much
so
we've
got
some
other
questions
that
have
popped
in
the
q
a
panel,
so
barack
td
want
to
take
a
look
at
them.
I'm
gonna
take
a
couple
right
right
from
the
get
go
for
the
first
one,
which
has
absolutely
nothing
to
do
with
with
delta
lake
how's
the
coffee
brewing.
I'm
yes,
I'm
currently
brewing
espresso
vivace
coffee
and
it
is
awesome.
So
I
believe
I've
now
answered
that
question
brock
td.
Do
you
guys
have
your
version
of
this.
B
C
So
can
delta
tables
be
created
on
the
glue
catalog
td?
You
have
a
little
more
context
here.
Potentially
I
would
say,
but
the
idea
here
is
that
you
know
spark
can
be
backed
up
by
the
glue
catalog
as
well,
and
you
know,
delta
can
work
in
such
a
way
that
you
can
use
the
glue
catalog
to
store
the
information.
C
But
the
caveat
here
is
that
you
know
the
information
that
delta
stores
in
the
catalog
or
the
meta
store
is
very
minimal.
It
just
stores,
like
I
said,
the
table
name
where
it's
located,
and
you
know
the
fact
that
it
is
a
delta
table.
So
through
the
glue
catalog,
you
might
not
see
information
about
what
the
schema
is
directly
from
glue,
which
is
kind
of
like
what
people
typically
care
about.
C
So
the
difference
there
is
like
can
spark
use
it
with
glue,
and
you
know
if
you
query
your
delta
tables.
All
the
time
from
you
know,
spark
using
this
glue,
metastory
yeah
sure,
but
do
you
get
like
the
full
experience
of
you
know
like
all
the
information
is
within
glue
and
you
can
take.
You
know
like
a
look
at
it
from
some
ui.
In
that
case,
no.
D
Yeah,
so
let
me
add
to
that
I
think,
basically,
theoretically,
it
should
absolutely
be
possible
for
delta
plus
spark
to
use
glue
actually
on
the
managed
delta
lake
on
the
side
of
databricks.
We
have
actually
made
that
work,
but
I
think
that
required
some
additional
work
to
make
spark
behave
well
with
the
glue-backed
metastore.
I
think
that
is
some
amount
of
work
that
probably
the
open
source
community
has
to
develop
before
it
can
actually
start
working
properly,
but
we
have
it
working
on
the
databrick
side
of
delta.
B
Perfect
and
then
I
think,
a
natural
progression
which
you
sort
of
already
covered,
but
basically
is,
can
we
query
the
metadata
to
find
the
details
from
databricks,
and
so
I
think,
there's
that
context
that
we're
saying
some
of
this
is
available
within
glue
itself,
but
some
of
it's
available
directly
within
to
the
transaction
log.
So
perhaps
cd
or
baroque
you'd
like
to
elucidate
a
little
bit
on
that.
C
Yeah
I
mean
since,
like
the
whole
idea,
is
getting
all
that
information
from
the
transaction
log.
Yes
within
databricks,
you
can
like
query
through
jdbc,
or
you
know
your
access.
You
know,
method
of
source
to
get
all
that
metadata
information
through
data
breaks
as
well
within
open
source.
Once
everything
is
hooked
up
in
the
correct
way
and
you
access
through
spark.
B
C
Right
question
so,
since
delta
has
to
you
know
like
compute,
its
states
and
the
state
computation
could
be
arbitrarily
expensive,
as
the
delta
table
gets
updated
and
the
state
is
computed.
That
state
is
cached
on
whichever
spark
cluster
that
you're
running
it
on,
but
any
time
you
access
that
delta
table.
C
It's
refreshed
and
this
refresh
mechanism
can
be
done,
synchronously
or
asynchronously.
It's
totally
up
to
you,
but
what
we
do
is
we
just
check
the
transaction
log
to
see
if
there
are
new
files
there
and
if
there
aren't,
then
we
can
just
go
ahead
with
the
cached
state.
If
not,
we
have
to
compute
the
cache
state.
B
Perfect,
perfect:
okay,
let's
continue
on
we'll.
We
actually
have
some
other
questions
here,
but
I
figured
let's
go
ahead
and
least
talk
about
some
other
stuff
and
then
we'll
go
back
to
some
of
the
other
questions
that
are
there:
okay,
all
right,
so
that
was
actually
just
the
first
slide
about
defining
tables
that
have
been
a
story
all
right,
oh
right
and
the
the
the
second
slide,
which
was
the
follow-up
on
that,
was
basically
here's
how
you
define
it.
You
can
define
using
the
data
frame
right,
save
as
table
name
object,
save
as
table.
B
Excuse
me
the
data
frame
write
two
or
data
delta
table
for
name
api
instances,
okay,
so
that
we
basically
made
it
easier
for
you
to
do
those
definitions.
Okay,
so
just
as
a
quick
call
out
slides
will
be
available.
We're
actually
referencing
the
document
and
on
me,
is
the
follow-up
blog
to
provide
details
on
this
stuff
as
well,
so
we're
gonna
make
sure
we
write
that
blog
and
get
that
out
as
well,
and
that's
td
looking
or
brock.
B
Looking
at
me
going,
you
know
you're
only
two
months
late,
so
yes,
my
bad
all
right.
So
the
a
cool
thing
absolutely
and
I'd
love
to
see
some
questions
based
off
of
called
delete,
updated
merge.
So
the
idea
is
now
that
you
can,
instead
of
just
using
the
pi
spark
python
apis
or
the
scala
apis,
you
can
actually
use
the
sql
ddls
okay,
so
you
can
run
delete
event,
so
you
can
run
updates.
You
can
run
merge
into
if
you've
already
been
using
delta
lake
on
data
bricks.
A
D
D
So
to
add
color,
basically,
the
merge,
sql
syntax.
We
support
in
delta
via
arc
data
source.
V2
now
is
one
of
the
most
advanced
merge,
syntax
out
there
in
the
community.
It's
even
more
advanced
than
the
hive
support
of
merge
there
is.
It
is
because
you
have
support
for
both
update,
delete
and
insert
clauses
which
is
beyond
the
standard
syntax,
but
high
supports
that.
But
what
beyond?
What
hive
can
support?
D
So
the
much
better
shorthand
one
of
the
most
heavily
used
extended
syntax
that
we
have
seen.
So
this
is
all
this
comes
with
spark
and
delta.
Only
and
I
don't
think
any
other
open
source
project
has
these
additional
syntax
that
makes
it
the
life
of
a
data
engineer.
Working
with
these
or
audit
analysis
is
much
much
easier.
B
B
B
So
I'm
I'm
guessing
the
for
the
person
who
who
asked
the
question
we
might
want
to
need
some
additional
details,
but
it
seems
to
me
that
what
you
want
to
do
is
you
want
to
be
able
to
just
use
the
cache
data
that's
stored
in
memory,
as
opposed
to
actually
going
ahead
and
looking
at
like,
for
example,
new
streams
or
new
batches
of
data?
I
think
that's
the
context.
B
C
C
What
that
does
is
that
you
know
like
if
you
have
queries
coming
in,
they
will
use
the
in-memory
cache
state
directly,
but
in
the
case
it
will
still
trigger
an
update.
You
know
underneath
the
hood,
and
hopefully,
if
your
you
know,
like
update
of
the
delta
state,
happens
within
those
24
hours,
you
will
still
continue
to
use
the
latest
state
of
the
table.
C
If
you
would
like
to
just
like
fix
your
state,
and
you
know
use
that
moving
on
what
you
can
do
is
you
can
define
a
view
and
leverage.
You
know
basically
time
travel
saying
that
I
want
you
know
to
select
this
table
timestamp
as
of
this
date,
and
that
way
you
can
actually
just
like
fix
yourself
to
a
specific
snapshot
of
the
table
during
that
day,
and
you
can
use
things
like
current
date
to
make
sure
that
you
always
get
like
today's
data
or
something.
B
Perfect,
perfect:
okay,
let's
continue
on
just
that
way
at
least
talk
about
some
of
the
other
cool
concepts
and
then
we'll
go
back
to
some
of
these
questions:
okay,
because
they're
com,
so
many
are
coming
through
so
td,
barack
as
as
we
continue
on.
If
you
could
take
a
look
at
some
of
the
q
questions
that
you
would
like
to
go
ahead
and
answer
ones
that
you
don't
want
to
are
not
appropriate,
obviously
dismiss,
but
most
of
them
are
pretty
cool.
B
So
obviously,
let's
just
try
to
organize
them
and
answer
appropriately.
Okay,
all
right!
So
actually
there
is
a
question
here
specifically
about.
Was
it
sorry,
I'm
just
trying
to
find
it
about
presto
and
athena
they're
good?
So
the
question
which
is
actually
a
good
timing
here,
so
do
we
still
need
to
define
tables
and
athena
presto
using
sim
links,
or
can
we
use
the
new
sql
method
of
defining
delta
tables
by
the
glue
catalog?
Okay?
So
we're
not
going
to
talk
about
the
glue
catalog
just
yet?
B
Okay,
because
I
think,
but
I
did
want
to
go
ahead
and
call
it
specifically,
this
context
automatic,
an
incremental
presto
and
athena
manifest
generation
so
prior
to
delta,
like
0.7.0
right.
What,
in
order
for
you
to
connect
your
presto,
athena
to
delta,
like
you
would
generate
the
manifests
of
a
delta
table
using
apache
spark
and
so
the
manifest?
Basically,
you
know
if
you
look
at
some
of
our
previous
tech
talks.
What
we
did
is
we
explained
the
concept
of
how
there's
a
transaction
log.
B
The
transaction
log
basically
tells
you
what's
the
latest
parquet
files
that
you're
supposed
to
be
reading.
Okay,
so
this
also
allows
us
to
do
time,
travel
basically,
data
versioning
things
of
these
cool
things.
Well,
what
is
the
manifest?
The
manifest
basically
is
a
file,
that's
generated.
That
tells
you
what
are
the
latest
files
for
you
to
work
with
so
that
ways
when
presto
and
athena
are
saying
hey
give
me
the
latest
delta
lake
information
we've
generated
manifest.
B
That
says
here
are
the
files
that
presto
or
theme
are
supposed
to
read
so
number
one
is
to
generate
those
and
then
number
two
is
to
configure
prestor
athena
to
read
those
generated
manifests
so,
in
other
words,
basically
read
a
list
of
files
that
says:
here's
the
parquet
files
you're
supposed
to
read
this
that
contains
the
latest
information
woohoo
all
right,
straightforward,
except
for
the
third
point,
which
is
how
about,
if
you're,
continually
updating
it
right.
Well.
Prior
to
this,
you
had
to
manually
update
this
information
right.
B
So
in
other
words,
you
had
to
go
back
and
say:
okay,
now
I'm
about
to
have
presto
query
the
stuff
query
my
delta
lake
table.
I
need
to
manually
update
that
yay,
except
no,
not
yay,
so
some
great
work
from
the
team,
which
is
to
basically
as
part
of
0.7,
to
do
that
automatically
and
so
you'll
notice.
Basically,
the
the
sql
syntax
alter
table
the
table
properties
and
then,
if
you
set
this
table
properties
to
the
same
sim
link
format,
manifest
isn't
able
to
true,
then
it'll
automatically
update
the
manifest
once
you've
generated.
B
D
C
D
It's
a
good
question,
but
it's
a
complex
question.
I
think
we
should
address
that.
So
so,
let's
separate
out,
let's
understand
what
is
going
on
underneath
and
separate
out
the
problems,
so
there
is
with
zero
seven
zero.
We
have
made
the
ability
to
define
and
put
the
minimal
metadata
needed,
to
associate
table
names
to
the
table
location
and
put
that
in
the
meta
store,
but
ultimately,
whether
you
can
use
that
metadata
to
query
the
delta
table
by
through
the
transaction
log
or
not.
D
That
is
something
that
needs
to
be
explicitly
built
in,
and
that
is
there
in
this
delta,
so
in
the
delta
implementation
inside
spark.
But
that
is
not
there
in
presto
athena.
Yet
so,
while
presto
athena
will
be
able
to
see
the
table
definition
in
the
meta
store
where,
if
you
connect
the
presto
athena
to
the
same
meta
store
as
you
are
using
on
the
spark
side,
so
you
will
be
able
to
see
the
table
definition,
but
it
wouldn't
know
what
to
do
with
it.
D
Using
it
wouldn't
know
you
how
to
use
that
metadata
to
read
the
transaction
log,
because
presto
does
not
yet
understand
the
concept
of
transaction
log.
So
yes,
while
it
is
visible,
you
cannot
yet
use
the
table
directly
in
athena
presto,
so
you
always
have.
You
still
have
to
continue
generating
that
manifest,
because
that's
what
athena
pressed
to
understand.
B
All
right,
perfect,
all
right,
we
looks
like
we
have
us.
We
have
continued
questions,
so
I'm
gonna
just
go
ahead
and
finish
off
the
slides.
We
only
have
a
few
more
anyway,
so
let's
go
ahead
and
do
those
and
then
we'll
go
ahead
and
go
right
back
to
the
questions
we
have
here.
Okay,
so
we're
gonna
go
ahead,
and
now,
with
these
particular
call
outs,
we
can
also
control
your
table
history
retention
right.
B
So
basically,
the
log
retention
duration
controls
how
long
the
history
for
our
tables-
okay,
don't
forget
with
delta
lake
with
the
time
travel
capability
right.
You
are
able
to
keep
the
history
of
your
table,
but
sometimes
you
want
to
be
able
to
control
how
long
the
logs
are
kept.
Ie,
the
transaction
log
and
how
long
the
actual
files
need
to
be
kept.
The
actual
parquet
files
all
right.
So
the
log
retention
duration
controls
how
long
the
history
for
that
table
is
kept,
I.e
the
logs
and
then
oh
and
then
the
delete
file
retention,
duration,
controls.
B
How
long
ago,
file
must
have
deleted
before
being
a
candidate
for
vacuum,
because
don't
forget
when
we
delete
we're
not
really
deleting
per
se,
it's
more
of
a
tombstone
what's
happening
is
that
we've
told
it
to
be
deleted?
We
know
the
information
is
like
in
there
now
there's
a
new
version
right
well
when
we
run
the
vacuum.
That
goes
and
clears
out
older
versions.
B
C
Yeah
one
common
like
misconception
I
want
to
add
here
specifically,
is
that
you
know
when
you
use
vacuum
vacuum
only
deletes
data
files
that
don't
that
are
not
tracked
anymore,
that
don't
have
tombstones
anymore,
the
log
files
themselves,
so
people
typically
look
at
the
log
directory
and
then
see
that
oh
here
are
files
that
are
older
than
seven
days.
Why
are
these
still
here?
They
ask
these
questions.
C
C
D
C
Yeah
fantastic
question:
so
when
we're
doing
time
travel,
we
don't
leverage
the
snapshots
time.
Travel
is
achieved
through
the
json
files.
The
individual
commit
information.
Therefore,
in
order
to
be
able
to
do
time,
travel
you
need
a
you
know,
compacted
parquet
file
and
all
the
json
files
after
it.
So
this
is
all
governed
by
this,
like
log
retention,
duration
setting
as
long
as
you
know,
your
log
retention
duration
setting
is
like
10
days.
For
example,
there
will
be
a
snapshot,
that's
at
least
10
days
old,
and
then
you
can
go
and
compute
something.
C
A
great
question:
it
really
depends
on
you.
I
would
say
how
much
you
want
to
pay
for
costs.
Typically,
like
I
mean
there,
shouldn't
be
any
like
upper
bound,
but
if
you're
having
updates
to
your
table
every
minute,
then
it
doesn't
really
like
it's
not
a
cost-effective
option
to
store
years
worth
of
logs.
C
B
Time
it
would
help
five,
I'm
muted,
myself
cool.
Thank
you
very
much
all
right.
Let's,
we
only
have
two
more
slides
to
dive
into
it
and
then
we'll
go
back
to
the
questions.
So
we
actually
have
lots
of
other
good
stuff
goodies
as
part
of
delta,
like
0.7,
the
support
for
adding
user-defined
metadata
and
when
you're
running
your
delta
table,
commits
support
for
adls
gen
2,
because
spark
3
has
support
for
hadoop
3.2
libraries.
B
This
actually
enables
basically
for
delta
lake
to
have
support
for
working
with
adls
gen
2.
Now,
okay,
so
for
all
of
our
azer
databricks
fans
or
for
that
matter,
azure
fans,
then
yes,
woohoo
now
we're
working
on
the
adl
too
also
improved
support
for
streaming
one-time
triggers
with
spark
3.0.
We
ensure
that
the
one-time
triggers
ie
trigger.once,
we,
which
we
often
use
in
examples
when
we're
talking
about
like
how
you
can
take
streaming
to
work
with
even
what
is
traditionally
seen
as
batch
based
processes
right,
but
but
it
works
really
well.
B
D
Yeah,
no,
no!
You
had
you
actually
said
it
correctly.
Just
I
think,
let
me
add
a
bit
more
context
on
people
who
may
not
be
familiar
with
trigger
ones.
Basically,
what
the
trigger
ones
mode
is
that
you
write
a
streaming
query
using
struct
spark
structure
streaming,
but
you
can
instead
of
running
that
streaming
query
continuously
24
7!
You
can
run
it
like
a
periodic
batch
job
where
you
say
that
start
this
streaming
query
in
trigger
once
mode,
which
will
essentially
do
only
one
trigger.
C
D
Process
all
the
available
data
data
and
then
shut
itself
down
so
that,
basically,
instead
of
if
you,
if
you
do
not
have
very
tight
latency
requirements,
you're
okay
with
few
hours
of
latency
or
one
day
of
latency,
you
can
run
the
same
streaming
query.
Instead
of
continuously,
you
can
run
it
like
using
airflow
or
something
triggered
externally,
triggered
one
time,
bad
job
to
process
all
the
data
new
data
available
and
then
shut
down,
and
then
maybe
six
hours
later,
12
hours
later
or
24
hours
later
run
it
once
again.
D
D
You
get
all
the
goodies
that
comes
with
structure
streaming,
but
you
are
essentially
a
lot
more
flexible
in
your
resource
cost
versus
latency
trade-offs
that
you
can
do
so
there,
the
you,
the
semantics
that
usually
is
desired,
is
that
when
you
do
run
this
trigger
with
this
one-time
trigger,
you
want
to
process
all
the
outstanding
data.
So
earlier
there
was
a
kink
where,
if
you
in
in
the
delta
data
source,
you
can
define
limits
that
in
each
trigger
or
in
each
micro
batch
how
many
files
to
process.
D
If
that
is
useful
for
24
7
operation,
you
don't
want
that
to
be
considered
when
you
want
to
do
trigger
once
operation,
because
you
want
to
process
all
the
austrian
data
so
that
kink
has
been
fixed
as
part
of
this
delta
zero,
seven
zero,
making
it
much
smoother
to
use
it
would
structure
streaming
in
this
trigger.
Once.
B
Cool
rock
on
all
right.
Well,
how
do
you
enable
all
this?
So
we
had
we've
talked
about
all
this
cool
stuff,
but
the
one
thing
that
we
want
to
make
sure
we've
called
out
is
that
you
actually
need
to
enable
data
source,
v2
and
catalog
api
integration
to
to
run
the
aforementioned
features.
So
to
do
that,
basically,
we
just
called
out
exactly
the
spark
session.
B
The
config
modifications
that
you
have
to
do
so
just
make
sure
you
do
that
when
you
create
your
new
spark
session,
you've
called
out
the
fact
that
you
are
enabling
the
sql
extension,
the
delta
spark
sql
session
extension
and
the
catalog
right,
and
then
once
you
do
that,
then
all
these
features
that
you
see
here
in
the
previous
slides
boom
now
you're
actually
able
to
go
use
them
so
pretty
cool.
I
don't
think
there's
anything
else
to
add
here,
but
that's
that's
about
it.
For
this
one
right,
tdm,
brock,.
C
B
Sweet
all
right.
Well
then,
let's
dive
into
questions
we
have.
We
still
have
plenty
of
questions
here,
we're
probably
about
10
minutes
worth
of
left.
I'm
gonna
start
with
the
first
one,
but
then,
if
you
guys
see
questions
that
you
like
to
answer
by
all
means,
let's
go
do
that
so
the
first
thing
I'm
going
to
call
out
is
like
what's
the
performance
impact
on
live
queries
when
a
vacuum
is
in
progress?
Okay,
so,
as
a
quick
call
out,
I
want
to
provide
context
for
anybody.
B
That's
you
know
relatively
new
to
the
context
of
what
this
is.
Vacuum
is
basically
deleting
the
older
files
right
just
as
per
that
log
retention
delete
direct
detention.
Duration.
Excuse
me,
the
context
is
that
we're
going
to
delete
when
you
run
vacuum,
we're
going
to
delete
all
the
files
that
are
you
saying
it's
older
than
seven
days
or
older
or
whatever
days
and
we're
going
to
remove
them.
Okay,
the
question
then
in
hand
is
that
will
that
actually
have
an
impact
when
I'm
trying
to
query
data
at
the
exact
same
time?
B
C
Yeah,
I
would
say,
like
the
cool
thing
here,
is
multi-version
concurrency
control.
That
just
says
that
you
know
like
with
multiversion
concurrency
control.
You
know
which
files
you
actually
have
to
query.
You
know
which
files
you
don't
have
to
care
about
and
with
this
separation,
like
vacuum,
is
entirely
its
separate
thing
and
it's
touching
separate
files
than
the
files
that
you're
querying,
even
if
they
are
like
it,
could
be
in
the
same
directories
like
neither
the
object
store.
Nor
delta
really
cares.
C
So
you
know,
like
you,
shouldn't,
observe
any
performance
implications
here
and
like
one
of
the
reasons
that
you
don't
observe.
Any
performance
implications
is
because
delta
doesn't
need
to
list
directories
or
list
files
to
figure
out
which
files
it
has
to
actually
read.
Vacuum
is
a
very
you
know,
object
store
heavy
operation,
both
the
entire
listing
and
the
deletion
operations
are
pretty
heavy
operations
on
the
object
store,
but
these
don't
affect
the
you
know,
get
performance,
for
example,
of
these
files,
specifically.
D
Okay,
one
assumption,
though-
and
this
is
a
semantic
assumption-
that
you
are
not
vacuuming
the
versions
that
you're
querying,
maybe
using
time
travel
so
say
you
are
vacuuming.
You
want
to
vacuum
every
version
that
is
more
than
one
hour
old
than
the
current
version.
Then
you
probably
should
not.
You
should
not
be
quitting
using
time.
Travel
any
version
older
than
one
hour.
So
as
long
that
semantic
guarantee
is
ensured
at
the
application
level
or
at
the
user
level,
they
will
not
interact
with
each
other.
C
B
Right
and
actually
to
exactly
to
these
points,
so
this
is
something
we're
guilty
of,
and
this
is
so
that's
actually
what
I
want
to
call
out,
which
is
the
fact
that
we
actually
have
a
lot
of
demos
where
we
actually
tell
you
vacuum
zero.
Now,
in
those
demos,
when
we're
running
vacuum
zero,
we
do
actually
make
sure
to
actually
put
comments.
Don't
ever
run
this
in
production.
Okay,
and
the
reason
we're
saying
is
basically
you
say
is
because
the
reason
we're
doing
in
the
demo
is
just
to
let
you
know.
B
Okay,
now,
you've
deleted
the
files
that
were
older
than
what's
right.
Now,
so
let's
say
you
have
10
versions
or
10
data
versions,
and
basically
every
single
version
has
been
deleted,
except
the
for
the
10th
version,
that's
great
and
then,
when
you
run
a
query
to
look
for
the
previous
version,
it'll
layer
out
as
appropriate,
because
the
data
is
no
longer
there
right.
So
that's
all
good
and
fun,
but
that's
the
reason
why
we
tell
you
yeah
like
exactly
to
td's
point
like
yeah.
B
You
want
to
be
careful
when
you
run
how
long
ago,
the
history
that
you're
running
your
vacuum,
because
if
you
do
yes
invariably
you're
gonna,
run
a
time.
Travel
query
at
the
same
time,
you're
deleting
it,
but
again
that's
not
that
common
and
the
most
commonly
played
out
is
is
exactly
the
way
we
described
it,
which
is
you
know.
The
date
is
seven
days
old,
okay,
yeah
we're
not
time
traveling,
seven
days
anything
older
than
seven
days,
then
we're
perfectly
cool,
but
just
again
just
want
to
call
that
out.
B
Okay,
all
right,
there's
actually
a
couple
questions
that
I
think
td,
you
would
love
to
go
dive
into,
so
I'm
going
to
call
you
out
right
from
the
get-go
is
basically
first,
it's
actually
related
to
structure
streaming
and
then
a
scenario
related
to
structure
streaming.
Okay,
so
the
first
question
about
specific
structure
means
in
structure
streaming.
Is
there
support
for
all
output
modes
for
delta
lake,
since
files
themselves
have
only
append
only
modes
and
so
or
excuse
me
at
pen
modes
right?
D
So
let's
say
you
have
a
streaming
query
that
is
computing
aggregates,
which
is
going
to
inevitably
generate
key
value,
pairs,
key
and
aggregate
pairs,
and
the
aggregate
will
keep
updating
you
if
you
want
to
write
or
essentially
upset
those
aggregates
into
a
delta
table.
What
you,
instead
of
update
mode
directly,
you
can
do
update
mode
for
each
batch
and
the
the
output
of
each
batch
will
contain
the
updated
aggregates
that
was
updated
in
that
micro
batch.
D
You
can
call
merge
on
top
of
that
on
that
batch
output
data
frame,
and
so
it
will
be
for
each
batch
batch
data
for
each
of
the
batch
data
frame
delta
table
dot,
merge
into
the
target
table
using
this
batch
output
data
frame
as
the
source,
and
there
are
examples
on
the
data
brick
on
the
delta
lake
docks
for
that,
so
you
should
be
able
to.
You
should
be
able
to
very
easily
copy
paste
that
and
start
building
on
top
of
that.
D
B
Perfect
so
then
related
to
exactly
that
dd
so
can
we
use
delta
lake
in
scenarios
where
the
tables
are
updated,
frequently
say
every
five
minutes.
Basically,
we
have
tables
stored
in
an
online
system
and
we
want
to
create
an
offline
system
using
delta
lake
and
update
the
delta
tables
every
five
minutes.
What's
the
perf
and
cost
implications
the
perf
and
cost
implications
for
doing
something
like
this.
D
Okay,
so
I
so
yes,
absolutely
you
can
do
that.
I
think
the
ambiguous
part
in
that
question,
which
I
don't
get
isn't
what
does
online
system
mean?
So
if
you,
if
you
are
talking
about
maintaining
online
system
like
a
database
like
so
a
common
pattern,
for
example,
let
me
use
a
patented
common
usage
pattern.
Is
there
is
a
tpcds
style
online?
D
Sorry,
there
is
an
oltp
style
online
databases
that
are
used
for
online
transactions
for
look
fast,
lookups,
point,
lookups,
etc,
and
often
people
do
is
that
in
order
to
do
larger
data
analysis
on
the
oil
on
the
oltp
data
without
actually
affecting
the
oltp
database,
people
take
change.
Data
from
the
well
db
database.
Take
the
change
log
stream
it
and
using
ups
using
merge
abserted
into
a
delta
table
corresponding
to
that
oltp
table
and
therefore
build
olap
kind
of
data
warehouse
in
the
delta
table
format.
That's
a
very
common
pattern.
D
We
have
seen
our
customers
as
well.
People
in
the
business
community
do,
and
so
basically
on.
You
have
essentially
the
same
data
that
is
sitting
otp
database
for
your
online
transactions
as
well
as
it
is
continuously
synced,
with
low
latency,
like
potentially
five
minutes,
absolutely
into
a
olap
style
data
warehouse
for
more
heavy
duty,
aggregation
trans,
that
kind
of
queries.
So
that
is
absolutely
possible
and
is
a
pretty
common
pattern.
B
B
What
we'll
do
is
we'll,
in
addition
to
writing
that
blog,
we'll
make
sure
to
actually
answer
as
many
questions
that
we
didn't
get
a
chance
to
answer
and
prop
that
into
the
the
faq
for
the
for
this
as
well.
Okay,
but
some
a
fun
question
just
because
I
I
also
do
the
same
thing
on
a
regular
basis,
even
though
it's
completely
unsupported,
because
brock
will
just
hit
me
for
doing
it.
B
Can
we
undo
a
specific
or
last
operation
in
a
delta
table
like
as
in
okay
and
manually,
deleting
the
json
for
the
underscore
delta
underscore
log
right,
but
this
is
probably
not
supported
and
probably
will
leave
your
your
transaction
log
in
an
inconsistent
state
so
barack
by
all
means
why'd
you
tackle
this
one
and
I'll
I'll
I'll.
Add
my
color
commentary
too
yeah.
C
Yeah
great
question:
we
do
not
recommend
you
tamper
with
the
delta
logs,
especially
do
not
ever
delete
anything.
You
know
like
don't
delete
an
intermediate
version
or
something
it's
against
delta's
protocol
to
overwrite
delta
logs.
It's
against
delta's
protocol
to
manually
delete
those
delta
logs
in
open
source.
The
easiest
way
to
you
know
like
undo
your
operation.
C
D
Hold
up,
oh
sure,
on
that
note
I
mean
when
you,
when
borac
mean
and
says
replace
the
data
in
the
table.
You
can
do
it
at
a
partition
level,
granularity
where,
if
you
know
that
some
operation,
that
affected
part
is
a
partition
of
the
entire
table
because
it
got
corrupted
or
incorrect
operation,
you
can
do
it
at
the
partition
level
generality.
So
that
is
you.
B
So
related
to
the
transaction
log,
just
because
a
lot
of
people
like
playing
with
it,
but
it's
a
good
question
by
the
way
if
the
parking
metadata
file
is
created
after
10
commits
now
remember.
This
is
the
call
out
that
after
10
commits
in
your
transaction
log
and
basically
the
10th
one
is
a
parquet
file
as
opposed
to
json.
B
Okay,
does
this
mean
I
can
only
go
10
commits
back
or
time
travel
queries
just
ignore
the
parking
metadata
files
and
so
I'll
quickly
answer
a
little
bit
but
barack
is
the
is
the
is
the
lead
on
this
stuff?
So
I
want
definitely
him
to
take
on
most
of
it,
but
no,
you
can
certainly
go
farther
back
than
10
commits.
So,
let's.
C
B
C
Yeah
exactly
I
mean
so,
the
parquet
files
are
written
after
the
json
files
are
also
written,
so
the
delta
files
must
be
written
first
and
then
the
parquet
files
are
just
an
optimization.
As
danny
mentioned,
to
speed
up.
You
know
the
state
computation,
so
you
can
definitely
go
behind
more
than
10
commits
it's.
C
Basically,
the
log
retention
duration
defines
that,
okay,
you
know
like
there
will
be
these
json
files
and
at
least
you
know
like
a
checkpoint
file
before
I
start
deleting
these
things,
and
what
we
can
do
is,
for
example,
go
back
30
days,
keep
a
checkpoint
file
and
keep
all
delta
files
after
it.
That
way,
we
can
always,
you
know,
like
compute
any
state
in
the
transaction
mode.
B
Perfect,
perfect,
perfect,
okay,
so
actually
we're
gonna
wrap
it
up
with
one
one
more
question
and
then
we're
gonna
wrap
it
up
again
again,
apologize
for
folks
that
we're
not
able
to
go
ahead
and
answer
your
questions
but,
like
I
said
for
those,
I'm
gonna
we're
gonna,
go
and
write
an
faq,
and
so
that
way
and
get
that
out
for
everybody,
so
we'll
have
the
blog
and
faq.
That
covers
a
lot
of
cool
stuff
all
right.
B
So
the
question
is:
what
can
we
not
do
in
0.7.0
that
we
can
do
in
databricks
delta?
And
so
I'm
going
to
actually
give
you
a
slightly
different
answer
than
you're,
probably
expecting,
which
is
what's
the
premise
of
what
we
have
in
in
the
delta
lake
in
databricks
versus
the
delta
lake?
That's
open
source,
so
the
context
from
where
we're
coming
from
is
that
we're
working
very
hard
to
get
delta
lake,
so
every
single
api
that
is
that
you
see
in
databricks,
is
also
available
open
source
and
the
other
way
around
right.
B
We
want
there
to
be
complete
and
utter
parity
between
the
two.
So
that
way
you
can
you,
anybody
can
use
delta
lake,
whether
they're
using
data
bricks
or
not
right.
It's
it's
an
open
source
project,
it's
part
of
the
linux
foundation
and
so
we're
working
rapidly
to
do
that.
Okay
and
we're
pretty
close,
but
you
know
we
still
have
some
work
to
do.
Let's,
let's
be
fair,
that's
why
it's
still
0.7
and
and
we'll
have
a
an
ama
in
in
october,
for
is
there
that
eight?
B
So
we're
not
we're
not
finished
yet,
but
we're
getting
pretty
close
right.
The
the
goal-
and
I
think,
if
you
ever
listen
to
anything
that
michael
armbrus,
would
talk
about
in
some
of
his
previous
videos.
You'll
hear
them
basically
very
clear:
the
apis
are
open,
source
they're
available
they'll
be
ever
it'll,
be
there
will
be
absolute
and
utter
parity.
B
There
will
be
some
things
like
performance
that
will
be
specific
to
data
bricks.
That's
true!
Okay,
because
we're
managed
service
and
we're
going
to
make
sure
it
runs
really
well
on
databricks,
but
it
doesn't
mean
you
yourself
cannot
do
the
same
thing
when
you're.
If
you
were
to
run
the
things
yourself
on
your
own
delta
lake.
B
For
you
know,
people
there's
actually
a
pretty
cool
question
in
terms
of
what's
one
of
the
cooler
projects
that
here
with
spark
well,
in
fact,
there's
a
pretty
cool
project
with
spark
and
delta
lake.
That's
actually
about
astronomy
data
to
actually
look
at
all
the
stars
using
those
super
powerful
telescopes
down
in
south
america
in
chile,
because
they
can
actually
go
above
the
the
cloud
base,
so
they
can
actually
see
everything
and
how
they're
processing
all
of
that
cool
data.
B
Well,
guess
what
they're
running,
obviously
in
open
source,
because
well
you
know
their
governmental
agency.
So
that's
what
they're
doing
right,
but
they're
running
on
delta.
For
precise
of
that
reason,
so
it's
we
want
to
make
sure
that
all
the
apis
are
there
for
everybody
to
use.
There
are
some
there
will
be
some
performance
improvements
within
databricks
itself,
but
overall,
what
you
see
open
source
and
what
you
see
in
databricks
will
be
the
same
from
the
standpoint
of
api
usability
and
functionality,
brock
td,
anything
else
that
I
missed
cool
all
right.
B
Well
then,
ladies
and
gentlemen,
I
will
go
ahead
and
flip
over
here.
I
did
want
to
remind
everybody
and,
first
of
all,
for
starters,
I
want
to
thank
td
and
barack
for
this
awesome
session.
It
was
and
thank
you
for
the
audience
here
for
you
attending
today's
session
again
apologize
for
not
necessarily
asking
answering
all
your
questions.
We
had
a
lot,
so
we
did
our
best
to
go
through
them,
we're
going
to
go
ahead
and
follow
up
with
an
faq.
B
Meanwhile,
we
have
some
great
data
and
ai
online
meetup
sessions.
That'll
be
coming.
This
thursday
will
be
a
glow.
A
wgr
genomics
panel
with
regeneron
and
databricks
it'll,
be
talking
about
how
we
can
do
g
was
at
an
amazingly
powerful
scale,
basically
because
of
glow,
wgr,
okay
and
then
the
following
week.
We'll
actually
have
two
fun
delta
lake
sessions,
the
first
one
with
douglas
moore
myself
on
generating
surrogate
keys
for
your
data
lake
house
with
spark
sql
and
delta
lake.
This
is
a
follow-up
to
our
scd
type
2..
B
We
had
a
lot
of
questions
about
surrogate
keys
when
we
were
doing
our
olap
style,
scd
type,
2
dimensions
and
how
you
did
that
in
delta
lake.
Well,
this
invariably
led
to
certain
key.
So
there
are
enough
questions
that
we
just
said:
hey,
let's
go
actually
create
a
session
specifically
on
that,
okay
and
by
the
way,
the
same
thing
as
we
hear
more
and
more
questions
from
you
in
these
q
a
panels,
even
if
we
don't
necessarily
track
all
of
them
or
answer
all
them
right
away.
B
We
will
make
sessions
based
off
of
your
feedback.
So
please
keep
on
chiming
in
here,
okay,
and
for
those
of
you
who
are
still
relatively
new
to
delta
lake.
Louis
is
actually
going
to
be
doing
a
tutorial
on
how
delta
lake
supercharges
your
data
lakes
on
thursday,
the
27th.
So
that's
it
for
at
least
for
the
next
10
days.
Okay
in
terms
of
three
different
sessions,
the
next
10
days,
so
we
hope
they
see
you
again
on
this
again.