►
Description
This talk is brought to you by the Istanbul Spark Meetup.
Abstract: This live coding session is a gentle introduction to the latest and greatest of Delta Lake (https://delta.io/).
You will learn what Delta Lake is and what challenges it aims to solve. You will hear about how Delta Lake builds upon the features of the recent Apache Spark 3 and why it can complement your data processing workloads.
During this talk, Jacek will talk about the slogan from the main page of Delta Lake: "Delta Lake is an open-source storage layer that brings ACID transactions to Apache Spark™ and big data workloads."
You will learn about time travel and data versioning using Spark tables in Spark SQL and Spark Structured Streaming. Databricks is proud to announce that Gartner has named us a Leader in both the 2021 Magic Quadrant for Cloud Database Management Systems and the 2021 Magic Quadrant for Data Science and Machine Learning Platforms. Download the reports here. https://databricks.com/databricks-named-leader-by-gartner
A
So
yazaki,
that's
my
that's
me:
I'm
I.t
freelancer
specializing
in
apache
park,
delta,
lane
and
kafka.
So,
if
you
need
me
for
anything
and
you
don't-
and
you
are
not
for
spark
that
the
lake
or
kafka
well,
our
conversation
may
be
a
little
bit
tricky
to
get
right
on
track,
but
so
just
just
think
about
these
three
projects:
spark
delta
lake
and
kafka.
A
If
you
need
to
hire
me
okay,
first
and
that's
that's
my
main
focus
for
the
past
five
years,
and
I
I
can't
imagine
any
other
day
without
them,
you
can
contact
me
at
theater,
pier
or
just
dm
on
twitter
at
yatsek,
okay,
I'm
best
known
by
internals
of
online
books
available
at
this
url
https
books,
yapila
pl
and
a
few
days
ago
I
started
yet
another
writing
journey.
Where
I'm
describing
my
experience
with
exploring
the
depths
internals
of
pi
spark.
A
I
found
it
quite
entertaining
especially
looking
at
python
code
from
java,
slash,
scala
and
perspective,
so
yeah
that
that
shows.
That
shows
me
different
spark.
Definitely
I
I
see
you
know
all
the
collaboration
between
python
pi
spark
play4j
and
the
core
of
spark
so
nice
to
see
this
extension
and
from
from
from
the
bottom
okay.
So
that's
about
me
friendly
reminder,
just
as
I
said,
if
you've
got
any
questions,
just
leave
them
in
the
chat
window,
but
you
know
it
might
be
that
I
will
be
able
to
answer
these
questions
earlier.
A
Definitely
expect
answers
at
the
end
of
the
dock:
okay,
two
okay
delta
lake,
so
homepage
delta
lake
io,
two
versions
of
delta
lake,
one
is
open
source
known
as
delta
lake
oss
open
source
software.
That's
the
aim
of
this
talk.
I've
got
no
idea
what's
available
in
databricks
commercial
delta
lake
version,
and
I
heard
good
stories
about
things
like
the
order
index,
which
is
not
available
in
oss
version,
which
you
may
and
you
can
find
in
data
bricks
quite
recently.
A
A
I
haven't
tried
it
yet,
but
well
it's
nice
to
see
them
on
gcp,
somehow
I'm
more
to
gcp
than
the
others.
No
idea
why,
anyway,
so
what's
delta
lake,
this
is
just
a
quote
from
the
website:
open
source
transaction
transactional,
storage
layer
over
object
stores.
I
hope
during
this
presentation
you
see
enough
demos
enough
examples
to
understand
what
they,
what
they
mean
by
saying,
open
source,
transactional
storage
layer
over
object
stores.
A
You
now
understand
why
they
say
open
source
because
well,
it
is
open
source
project
and
transactional
storage
layer.
It's
coming
of
over
object
stores
well
in
the
cloud
object
stores
are
the
storage
so
well.
It's
obvious
that
targeting
object
stores
with
something
transactional
may
have
some
traction
and
yeah
delta
lakes
seem
to
attract
a
lot
of
attention.
A
As
I
mentioned
two
hours
ago,
I
had
conversation
with
a
10-ish
group
of
people
from
mostly
from
poland,
large
organizations,
banks,
financial
institutions,
and
they
were
all
into
a
spark
and
delta
lake.
So
yeah,
that's
nice
to
see
this
growing
adoption
as
in
transactions.
Yes,
it
is
available
as
part
of
delta
lake.
That's
basically,
what
delta
lake
provides
mainly
all
other
features,
all
other
goodies.
A
One
may
consider
site
effect
of
doing
asset
transactions
for
spark
so
features
like
time
travel.
Well,
if
you
are
dealing
with
transactions
you
may
cons,
you
may
imagine
some
notion
of
versions
of
your
underlying
object,
store
or
storage,
and
that's
what
happens
in
delta
lake
delta
lake
keeps
track
of
all
the
versions
transactionally
or
in
the
transaction
by
transaction
and
keeping
track
of
all
these
versions.
A
Give
gives
you
automatically
this
time
travel
feature,
so
you
can
just,
for
example,
switch
from
latest
version
of
your
table
or
your
delta
table
to
something
that
was
available
in
the
past
in
version
zero.
Eight,
they
added
support
for
streaming
from
the
latest
version
of
your
table,
so
for
some
people
more
familiar
with
kafka,
it
may
seem
like
reading
from
delta
lake
is
pretty
much
like
reading
from
kafka
topic
and
yeah.
I
I
found
it.
A
There
is
this
feature
of
scalable
metadata
handling,
because
if
you
are
dealing
with
transactions
modifying
underlying
data
and
automatically
moving
to
another
version,
you
may
expect-
and
it
is
available
in
that
lake-
to
have
something
like
schema
enforcement,
making
sure
that
your
data
match
what's
expected
by
shape
by
column
names
and
by
the
types
as
well
as
allow
for
something.
That's
called
schema
evolution,
something
that
you
may
have
found
in
file
formats
like
avra
used
for
in
kafka
based
solutions.
A
What's
nice
about
delta
lake,
and
that
was
the
main
reason
why
I
looked
at.
Why
I
immersed
into
delta
lake,
was
that
it's
its
aim
is
to
unify
streaming
and
batch
data
processing
in
apache
spark,
and
this
is
just
yet
another
data
source
or
data
format
in
apache
spark.
So
for
many
people
looking
at
delta
lake
well,
it
just
opens
new
doors
to
new
features
that
are
not
available
in
apache
spark,
but
because
delta
lake
extends
what's
available
in
spark
by
spark
compatible
ways.
A
Well,
it's
just
kind
of
like
part
of
the
spark.
So
for
me
it
was
nice
to
open
my
my
understanding
of
what's
possible
in
big
data
world.
Without
you
know
joining
you
know,
high-priced
college
or
universities
that
I
probably
would
not
be
able
to
to
join
because
of
my
low
rank
or
something.
So
I
I'm
glad
that
big
brains
from
databricks,
you
know
coined
or
created
spark
and
now
delta
lake,
and
let
me
learn
from
them
by
looking
exploring
the
source
code
because
well
it's
all
open
source.
A
Okay.
So,
let's
start
with
demo
the
first
demo,
where
I'm
introduce
you
to
installing
delta
lake
in
your
spark
application.
By
saying
your
spark
application,
I
mean
spark
shell,
because
I'm
not
gonna
create
a
brand
new
spark
application.
It
would
take.
You
know
too
long
for
such
a
short
meet
up.
So
I
just
took
a
spark
shell.
I'm
gonna
take
spark
shell
and
just
extend
it
with
delta
lake.
I'm
gonna
show
what
it
what
it
takes
to
to
do
this
and
what
are
the
steps,
then
I'm
gonna
create
a
delta
table.
A
I'm
still
not
convinced
whether
delta
table
should
be
written,
lowercase
or
uppercase
yeah.
I
know
it
doesn't
really
matter,
but
still,
while
I'm
into
writing,
it's
very
important
for
me,
or
I
pay
attention
to
this
to
this.
To
details
like
this
and
but
anytime,
I
write
delta
uppercase.
It's
like
thinking
about
this
delta,
a
lines
even
though
it's
not
that
popular
in
europe.
A
Yet
whenever
I'm
in
us
thinking
about
delta,
it's
like
come
on
delta
lake
or
delta,
a
lines
I'm
digressing
again
so
and
then
I'm
gonna
show
how
to
read
and
write
to
from
oh
from
to
a
delta
table
again.
I'm
gonna
do
this
in
batch,
using
spark
sql
and
in
streaming
using
spark
such
as
string.
Okay,
ready,
buckle
up
and
let's
see
the
action.
A
A
Yeah
see
people
noding
their
heads.
Thank
you.
So,
first
of
all,
let
me
prepare
the
ground
because
you
know
I
may
have
done
some
presentations
before
so
just
to
make
sure
that
everything
is
okay,
I'm
removing
entire
file
system.
No,
no!
I'm
kidding
well,
every
time,
I'm
doing
all
these
presentations,
I'm
thinking
about
using
virtualized
environments
like
docker
or
kubernetes.
No
docker
would
be
enough,
but
I'm
not
that
adventurous
and
that
definitely
would
not
help
that
much
just
minus
rf
tmp.
That
should
be
fine.
A
Unless
you
make
a
mistake
and
you
put
a
space
between
first
is
slash
and
the
rest
no
matches
very
good,
because
that's
exactly
what
I
expected,
because
I
clean
it
up
before
I
meet
up,
but
just
to
make
sure
okay,
I
do
this
again,
okay,
so
what
I
did
first
of
all,
I'm
gonna
use
this
location
for
all
delta
tables.
Okay,
except
for
delta
tables
that
are
going
to
be
registered
in
this
hive
like
metastore,
that's
under
spark
warehouse
and
metastore
db.
A
Okay,
we
have
db
log
being
logs
of
this
relational
database
system
that
that's
that's
behind
this
meta
store
underscore
db.
That
supports
this
spark
warehouse.
Okay,
basically,
that
that's
what
happens
and
that's
why
I
remove
it?
Okay,
so
that's
it
now!
Let
me
show
you
how
to
start
how
to
install
delta
lake.
Well,
it's
as
simple
as
any
other
spark
package.
It's
just
minus
minus
packages
with
these
coordinates.
A
A
This
is
just
something
that
you
can
use
to
inject
anything
well,
not
actually
inject
extensions.
To
make
spark
sql
look
completely
different
from
what
spark
sql
experts
could
expect
or
knew,
and
so
I
warn
you
not
to
abuse
you
well
by
definition.
Abusing
is
something
bad
but
be
careful
what
you're
doing
while
working
with
spark
sql
extensions,
because
well
you've
got
full
access
to
all
the
optimization
layer,
all
the
anal
analyzes
phase,
all
the
phases
of
your
query
execution-
and
this
is
basically
available
through
this
spark
sql
extensions.
A
This
is
crucial
for
delta
lake
to
work
properly,
with
spark
to
interact
properly
with
spark
because
of
this
second
configuration
option,
which
is
spark
sql
catalogues
by
catalog.
This
is
the
name
of
the
default
catalog
in
spark
sql,
and
by
doing
by
installing
this
extension
now
you
are
telling
spark
sql
to
use
delta
catalog
to
be
the
first
whenever
any
table
is
looked
up.
So
first,
it's
checked
whether
it's
that
table
and
then,
if
all
is
fine,
well,
a
delta
like
delta,
like
and
delta
catalog
will
pass
it
down
the
chain
okay.
A
A
You
know
unsafe
platform
to
keep
all
the
data
outside
jvm
just
to
keep
your
garbage
collector
less
busy,
so
our
applications
become
faster
and
some
other
things.
So
you
will
see
this
warning,
but
you
can
just
disregard
this
them
these
warnings
and
then
loading
because
we
are
using
minus
minus
packages.
So
all
this
loading
happens
here.
I've
got
this
downloaded
already,
so
artifacts
number
attempted
to
be
downloaded
well
too,
but
none
was
really
downloaded
because
it
was
already
available,
but
definitely
it
was
resolved.
All
is
fine.
A
Okay,
now
we
are
in
spark
shell.
This
is
version
three
one,
one
release
candidate
three,
I
just
build
it
for
my.
You
know,
voting
purposes,
checking
before
I
vote
yes
or
no
non-binding
and
so
scala
212.
I
did
not
attempt
to
13,
even
though
partial
support
is
available,
it's
on
opengdk,
64-bit,
java
11
and
that's
pretty
much
it
yeah,
that's
pretty
much
it!
So
that's
about
my
environment!
A
A
Now
I
write
it
and
very
important
thing
is
that
by
default,
in
spark
parquet
is
the
default
file
format.
We
want
to
use
delta
lake,
so
we
specify
delta
as
the
default
as
the
data
source
to
be
used
for
saving
our
dataset.
Our
query
our
result
of
our
query
so
format
and
then
the
last
but
not
least,
save
obvious
right
and
delta
and
zero
one
and
that's
pretty
much
it.
A
If
I
typed
all
all
correct
all
correctly,
we
should
get
this
nice
messages
printed
out
to
the
console,
and
I
did
all
I
did
instruct
spark
shell
to
print
out
all
these
messages
on
purpose,
because
well,
that's
the
purpose
of
our
meetup
to
learn
more
about
delta
lake,
not
from
you
know,
reading
documentation,
but
by
looking
at
the
internals
of
delta
lake
and
looking
at
what
it
can
tell
you
by
increasing
lock
level
or
increasing
decreasing.
Depending
on
how
you
look
at
it.
A
I
definitely
enabled
all
dental
related
locks
to
all
level,
so
we
see
everything
that
may
ever
get
printed
out
by
delta
lake,
so
this
is
on
info,
so
you
will
see
it
by
default
because
by
default
info
is
the
default
lock
level
in
spark,
and
then
you
see
something
like
creating
initial
snapshot.
Well,
we'll
get
to
it
without
metadata,
because
the
directory
is
empty.
A
So
for
for
some
basic
description
of
delta
lake,
you
may
you
may
easily
and
safely
say
it's
parquet
plus
json,
encoded,
metadata.
Okay,
then,
so
this
is
delta
log.
This
is
the
front
end.
This
is
programmatic
api
that
you
can
use
from
your
applications
if
you
want
to
go
lower
and
below
a
high
and
high
level
operators
and
like
format,
save
and
so
delta,
lock
is
responsible
for
managing.
You
know
access
to
this
extra
directory.
A
We
are
going
to
look
at
in
a
sec.
Then
initial
snapshot,
let's
skip
it,
that
luck
again,
reporting
us
that
no
data
log
found
yeah
true,
because
well
this
actually
was
just
empty.
So
it's
it
should
come
with
no
surprise,
and
you
see
that
there
is
this
mysterious
battery
underscore
delta
underscore
lock.
A
As
I
mentioned,
I
just
started
exploring
pi
spark,
and
I
know
that
underscores
are
special.
It's
a
underscores
or
thunders
or
specials
in
special
characters
or
have
special
meanings
for
private
values
methods.
So
in
this
context,
it's
just
to
denote
that
this
directory
is
managed
by
by
delta
lake.
So
just
as
it
is
in
how
do
they?
How
do
they
say
in
english?
They
say
something
like
putting
horses
before
cart
or
something
like
this.
That
was
something
like
this.
A
There
was
the
same
so
just
as
you
know,
describing
something
that
that
should
be
introduced
later.
I
just
I'm
just
saying
I'm
just
going
to
say
that
they'll
underscore
delta
underscore
log
is
just
extra
directory
that
makes
this
directory
no
longer
pocket
only
okay,
so
you
could
read
this
directory
as
parket
and
it'd
be
fine.
A
Let's
do
this,
even
though
it
was
not
supposed
to
do
it's.
It
was
not
supposed
to
be
done
this
way,
but
let's
read
format
the
explicit
that
we
are
reading
parquet
and
load
n,
tnp
delta,
zero
one
and
show-
and
I
expect
to
see
exactly
what
I
saved
range02
gives
me
or
gave
me
a
two
record
data
set.
It
was
written
as
delta.
So
now
your
question
could
be
well.
Okay.
A
But
that's
where
this
similarity
ends
once
you
start
modifying
your
delta
table,
you
are
adding
features
that
are
not
supported
by
parkit
itself,
not
by
this
parquet
file
format
by
by
but
by
this
packet
data
source
in
spark.
Okay,
so,
and
that's
the
beauty
of
that,
okay,
so
would
things
change
if
we
did
this
whoops
sorry
spark
read
format,
delta
load
and
now
I'm
gonna
copy
it
same.
You
see
well,
except
these
two
entries.
A
Well,
you
could
just
disregard
them,
but
they
are
telling
you
something
about
delta
right.
So
there
is
this
filtering
files
for
query
and
done
just
to
make
your
you
know
query
faster,
because
basically
it
selects
our
front
table.
Okay.
So
now
you've
got
all
this
and
there's
this
notion
of
snapshots
of
your
tables,
because
well
because
of
multiple
files
sitting
behind
a
particular
version
of
your
table.
A
Well,
it'd
be
nice
if
there
was
a
mechanism
to
keep
this
information
handy
so
future
queries
over
this
particular
version
of
your
build
table
way
quicker
than
just
scanning
files
over
and
over
again
that's
what
this
snapshot
is
for.
It
keeps
a
track
and
all
this
information,
it's
ca.
It's
a
cached
version
of
your
delta
table
and
basically
it's
a
snapshot
of
your
belt
table
at
a
particular
version
and
what's
more
interesting,
is
that
we've
got
this
hold
on
hold
on.
A
We've
got
this
oops
and
we've
got
this
console
right
and
now
you
can
see
view
I
know,
and
now
you
see
well.
This
is
a
brand
new
web
ui
from
three
one
one,
and,
to
be
honest,
I'm
surprised
to
see
you
know
this.
This
ui
revamped
look
and
feel
like
or
wise
look
at
this.
The
this
this
is
smoother
in
a
sense.
A
It's
definitely
different.
It's
brighter,
I
don't
know,
can
describe
it
properly,
but
definitely
it's
different
from
what
I
got
used
to
seeing
in
previous
releases,
okay
and
need
to
train
my
eyes
to
to
get
accustomed
to
this
accustomed
to
this
coloring
scheme.
Okay,
so
under
sql
you
see
that
you've
got
this
show
twice
well
save
this
is
the
the
very
first
query
when
I
set
this
okay,
let
me
do
this,
so
we
can
just
so
when
I
did
this
remember
when
I
wrote
this
two
row
data
format
to
this
directory.
A
We
basically
did
this.
This
save
and
this
safe,
not
sure
why
we've
got
two
saves.
One
is
for
this,
probably
because
we
are
a
local.
This
is
local
collection.
That's
why
we've
got
this
safe
here,
because
it's
execute
this
is
command
that
that
was
executed
on
the
driver,
because
spark
range
creates
local
relation,
so
basically
there
is
no
distributed
computation
needed.
So
that's
why
you
got
this
extra
query,
but
why
is
it
shown
like
this
anyway?
Okay,
so
then,
you've
got
the
save.
A
A
Took
the
step
extra
step
to
be
ready
with
the
latest
version
of
delta
table,
you
just
saved
in
case
you
need
to
read
from
it.
You
know
in
the
same
application.
It's
just
assumption
that
it
may
be
needed.
Okay
and
since
the
latest
version
zero.
Well,
we've
got
it
snapshotted
and
snapshot
and
just
to
prove
my
point
that
it's
cached
data
lake
table
by
rdd
cache
or
data
frame
cache
you,
you
may
switch
to
storage
and
see
this
delta
table
version
here,
as
rdd
name
delta
table
state
zero.
A
This
is
the
version
and
this
is
the
location
of
your
delta
table.
Everything
is
cached,
so
nice
size
in
memory.
Okay.
So
it's
ready
one
replicated
this
memory
serialized.
This
is
the
default
storage
storage
level.
If
I'm
not
mistaken,
that's
that
that's
the
name
in
data
frame
and
yeah.
Basically,
you
can
just
look
at
it
and
and
50
partitions
and
yeah.
You
can
just
explore
it.
Okay,
we
are
again
we
are
looking
at
the
very
bottom
of
what
happens
under
the
covers
of
this
nice.
Looking
spark
range,
sprite
format.
A
Okay,
then
we
did
show
and
show-
and
that
was
this
delta
filtering
files
for
query,
well,
something
we
we've
seen
already,
but
I'm
surprised
it's
separate,
sql
query
anyway.
I
think
that
they
change
it.
The
last
time
I
looked
at
it
was
zero.
Seven.
I
haven't
spent
much
time
on
zero,
eight,
yet,
and
so
so
I
cannot
answer
whether
it's
something
new
in
zero,
eight
or
it
was
already
available
in
zero.
A
Seven,
I
don't
remember
it
having
it
seen,
okay,
so
that
was
how
to
install
delta
lake
and
now
create,
and
so
let's
move
on
to
reading
and
writing.
Okay,
even
though
we
did
something
partial
in
this,
you
know
space
yeah,
so
review
console
logs,
reviewing
console
logs
done
at
least
partially
we'll
look
at
it
again
soon
and
now,
let's
have
a
look
at
this
extra
directory
that
was
created,
something
that
we've
seen
is
name
that
that's
named
by
minus
delta,
underscore
lock
so
ls
and
tmp
delta
and
okay.
A
This
is
the
directory
where
we,
which
we
use
to
save
our
packet.
Well,
parket
delta
table
okay.
So,
as
I
showed
you
already
well,
this
is
just
basically
parkette
data
market
data
data
source.
I
don't
know
how
to
properly
name
it,
but
anyway,
this
is
just
definitely
a
data
frame
data
data
set
or
data
set.
That's
what
I
like
the
most.
So
this
is
a
parker
data
source
with
data.
As
I
said,
three
files-
fine,
okay,
now
we've
got
this
extra
directory.
A
Whenever
you
see
this
directory,
this
is
in
the
top
level
directory
of
your
delta
table.
You
immediately
know
that
you
are
dealing
with
delta
lake
table
okay,
so
whenever
you
see
underscore
delta
underscore
lock,
no,
you
are
dealing
with
delta
lake
table
okay.
So
let's
have
a
look.
What's
inside
this
delta,
lock,
which,
by
the
way
it's
called
transaction,
lock?
A
Okay,
the
very
first
time
I
saw
this,
it
reminded
me
the
days
when
I
was
looking
at
the
sources
of
kafka
and
how
all
the
you
know,
partitions
were
stored
on
disk
they've,
been
kafka,
uses
similar
notation
for
the
chunks
and
it's
been
a
while,
since
I
worked
with
kafka
internals,
so
I'm
not
sure
how
they
exactly
and
name
all
the
pieces.
These
chunks
of
data,
but
definitely
they
use
a
similar,
notation
or
format.
Okay,
then
I
realize
it's
similar
in
checkpointing
or
in
other
data
sources.
A
A
Then
next
version
is
gonna,
be
zero,
zero,
zero,
blah
blah
blah
one
dot,
json
two
blah
blah,
and
so
I
don't
know
what
would
happen
if
we
name
it
99999
and
then
just
you
know,
change
this
delta
table
and
attempt
it
to
to
save
it
as
version
I
don't
know
billion
or
billion
billions
whatever
so
yeah
I've
never
tried
it
myself,
yeah.
Let
me
know
if
you
did
so,
let's
see
what's
inside
so
look
at
this.
A
This
is
commit
info,
some
extra
information
that
you
may
or
may
not
have
with
some
extra
information
like
timestamp.
What
operation
created
this
transaction
commit?
It
was
batch
right
all
the
streaming
and
batch
operations.
Have
they
names
so
batch
right
is
right.
Then
there
are
operation
parameters
like
mode
error
if
not
exist.
This
is
the
default
safe
mode
in
spark
sql
partition
by
empty
we've,
we've
used,
no
partitions
is
blind
append.
A
So
if
you
are
adventurous
enough
and
you
wanna
and
you
want
to
interact
with
delta
lake
by
using
this
meta
store,
well
feel
free
to
go
and
parse
all
these
jsons
and
just
combine
them
together
and
well.
You
have
all
you
need
and
then
protocol,
and
that
was
changed
a
little
bit
in
zero
eight
because
they
added
some
extra
information
over
this
protocol.
I
can't
remember
what
it
was
exactly.
I
just
describe
it
in
my
notes
about
delta
lag,
but
it's
been
couple
of
days,
and
so
I
simply
forgot
metadata.
A
This
is
a
id
of
the
query
format,
provider
parquet,
which,
by
the
way,
is
not
changeable
yet
or
there
is
a
comment
that
fix
me
if
it's
gonna
be
ever
possible,
some
other
options
schema
string,
something
that
you
can
use
to
understand.
What's
the
shape
of
this
initial
version,
partition
columns,
other
configuration
options
and
create
a
time
which,
by
the
way,
should
be
similar.
Timestamp
of
this
commit
info
is
different
from
created
time
interesting.
A
It
might
be
because
this
metadata
is
about
transaction
when
it
was
started-
and
this
is
when
it
was
committed-
it's
possible,
but
I'm
not
sure
about
it.
Okay,
then
these
actions
or
actions
section
follows
so
we've
added
three
files,
just
by
looking
at
num
files.
Three,
we
know
that
we
are
dealing
with
three
files
in
this
version
and
now
we
know
that
these
three
files
were
all
added,
so
they
are
brand
new
and
we've
got
the
names
okay.
A
So
now
we
can
combine
this
information
information
from
this
transaction
log
and
use
this
to
construct
or
reconstruct
particular
version
of
our
delta
table.
Well,
this
notion
of
version,
even
though
I'm
I'm
using
it
already
and
for
some
time-
and
you
might
not
be
familiar-
you
know
what
exactly
is
version
in
delta
lake.
A
A
A
A
A
Thank
you
del
play.
Thank
you.
Randomly
watch.
If
you
have
any
questions,
welcome
a
little
off
topic,
okay,
a
little
of
topic,
but
is
scala,
used
much
in
production
with
kafka
or
is
it
java?
Oh
interesting
question,
which
I
found
a
little
relation
to
our
main
topic,
which
is
delta
lake
right,
but
anyway
yeah
a
little
of
topic,
so
scala
much
production
with
kafka,
not
sure
I
would
consider
java,
because
well
kafka
decided
to
switch
to
java
based
api
for
backward
compatibility,
opening
the
doors
to
to
java
devs
more
widely,
so
to
speak.
A
Even
kafka
streams
started
with
java
api
before
they
they
added
scala
api,
so
yeah.
I
I'd
not
be
surprised
seeing
more
java
devs
than
scala.
How
is
performance
matrix
and
consistency
for
asset
transaction?
It's
coming!
Well,
it's
just
optimistic
transaction!
So
well,
if
you
manage,
if
transaction
managed
to
write
this
json
file
to
transaction
log,
well,
it
just
won
this
race
for
saving
files,
so
you
will
see
some
files
being
created
and
and
and
then
recreate
it
while
transactions
are
retried.
A
A
We
are
speaking
about
different
volumes
right,
so
that
might
be
something
that
I'd
say:
don't
do
that
yeah.
Definitely
thank
you
could
not
might
have
troubles
with.
You
know.
Big
large
data
sets
that
that's
that's
where
I
would
I
I
would
not
be
surprised,
posgust
handled
poorly,
but
overall
other
than
that,
no
idea.
A
Okay,
so
I
leave
it
here,
okay,
so
let's
move
on
right,
and
so
we
reviewed
our
reviewed,
our
transaction
log
and
now,
let's
read
from
our
spark
table:
okay
spark,
read
format
and
delta
and
load
and
something
that
we've
seen
already
right
and
show
okay
and
false.
Okay.
Is
this
what
I
wanted
to
show
you
delta
yep,
so
pop
this
false
is
just
truncate
equals
false?
Okay,
as
a
matter
of
fact,
I
don't
need
this
okay,
so
we've
got
this
okay,
now
a
spark
table
yeah.
A
A
Okay,
so
let's
assume
we
have
two
two
spark
applications
up
and
running
okay,
so
this
is
our
initial
application.
Initial
spark
shell
and
this
this
is
yet
another
spark
applications
workshop.
Okay.
Now
what
I'm
gonna
do
is
to
import
yeah
import.
No,
not
this
one,
not
this
one
trigger,
and
the
reason
for
this
is
that
I'm
gonna
read
using
a
streaming
query
and
show
you:
you
know
how
sparks
access
streaming
deals
or
works
with
delta
tables.
Okay,
so
I'm
using
this
trigger,
and
now
I'm
okay.
A
Let
me
let
me
see
if
I
can
copy
it
yeah.
I
can
so
why
not
to
do
this
other
than
yeah
roof,
so
what
I'm
gonna
do
is
spark
with
stream
just
to
open
or
just
to
be
open.
This
declarative
streaming
a
way
of
describing
my
string
computation
seeming
distributed
computation,
I'm
saying
that
my
source
is
delta
delta
lake,
I'm
loading
data
from
tmp
delta,
0,
1
directory
and
then
after
I
load
it
after.
A
I
created
this
initial
starting
point
in
my
distributed
cyclic
graph,
and
I
write
this
stream
creating
end
point
for
my
data
and
I'm
using
format
console
just
to
print
out
all
the
data
I
get
from
this
delta
lake
table,
okay
and
I'm
doing
it
every
five
seconds.
That
was
the
reason
why
I
imported
this
trigger
and
that's
pretty
much
it
so
I
start
this
streaming
query.
A
A
A
A
Two
and
three
then
write
format
delta,
make
sure
that
you
are
including
format
delta.
Otherwise
you
will
deal
with
exceptions
that
are
far
from
delta,
related
mode
append.
We
are
appending
to
existing
table
and
then
you
know
the
directory
of
our
table
right
right.
So,
let's
save
it
and,
let's
see
what's
gonna
happen
here
hop
so
I'm
save.
I
save
this
these
two
rows
and
look
at
this.
A
A
So
now,
look
at
this
look
at
this
locks.
So
let
me
show
you
this
logs,
so
I'm
saving
two
records
and
now
optimistic
transactions
started
with
table
id,
which
was
something
that
was
available
in
meta
meta
in
this
transaction:
walk
attempting
to
commit
version,
one
with
four
actions
with
syria's
serializable
isolation,
isolation,
level,
loading
version,
one
okay
version
one
I
was
supposed
shouldn't-
be
zero
anyway,
so
table
compute
snapshot.
A
There
is
for
version
one.
Then
there
are
some
extra
information
like
created
delta
log
file
index,
it's
in
json
format
with
these
extra
files.
Oh
now
I
got
it
in
kafka.
They
call
it
segments
for
data
files
that
hold
data
on
on
disk,
so
segments
seem
very
similar.
Kafka
segments
seem
very
similar
to
this
numbering
scheme
in
a
transaction
log
file,
size,
extra
information
snapshot,
okay,
some
some
other
information,
and
then
you
know,
I
think
this.
This
transaction
was
attempted
to
be
committed.
A
It
succeeded,
it
was
okay,
committed
delta
hush
one
to
file,
and
that's
it
okay.
So
let's
have
a
look
what's
inside
this
directory
again
so
tmp
delta01
and
now
look
at
this.
You've
got
now
six
files,
or
so
it
depends
on
your
environment,
number
of
cpus
and
all
this
okay.
It
doesn't
really
matter.
Okay,
what
matters
is
what's
written
here
now.
You've
got
two
versions.
A
A
A
A
I'm
thinking
if
there
could
be
any
other
answer,
will
it
will
not
work
or
you
will
get
all
files?
Okay,
let's
have
a
look
it's
going
to
work
because
we
made
sure
that
all
versions
are
of
the
same
shape
of
the
same
schema,
so
mark
and
mitra
are
were
all
okay
were
all
correct,
but
there
were
some
assumptions
to
be
made.
A
A
Well,
because
there
is
no
notion
of
versions
right,
it
only
makes
sense
to
answer
these
questions
for
delta
lake,
because
well
delta
lake
supports
versioning
so
well,
yeah,
that's
the
beauty
of
delta
lake
and
that's
the
main
reason,
the
main
difference
between
parquet
and
delta
lake
formats,
even
though
they
share
much
okay,
so
nice.
So
let's
have
a
look.
What's
going
to
happen?
Oh
let
me
read
it
again.
What's
going
to
happen
if
I
change
this
format
to
delta,
and
now
I'm
asking
you
what's
going
to
happen
now.
A
For
different
schemas
is
it
possible
to
block
reading
delta
with
parquet
on?
I
don't
think
so.
No,
I
think
it's
the
default
mode
on
databricks,
really,
okay,
to
my
to
my
understanding
to
my
knowledge
of
spark
and
delta
lake
I'd
say
it's
not
possible
to
limit
what
you
can
do
with
a
particular
data
source.
A
While
reading
data
from
directory
you've
got
access,
you've
got
reading
access
too
so
yeah.
I
I
can't
think
of
a
mechanism
that
would
prevent
you
from
doing
this
so
yeah.
I
I
think
it's
not
possible
to
block
reading
delta
with
parquet
I'd,
be
it'd,
be
very
nice
to
to
hear
I'm
wrong
because
I
could
learn
something
from
here.
Oh,
I
think
there
is
an
option.
Thank
you.
Thank
you,
pablo
okay,
adding
for
different
schema.
You
will
get
all
files
from
different
schema
earnings,
not
sure.
I
understand
this
question
for
different
schema.
A
Actually,
you
get
an
error
if
you're
trying
to
read
it
as
a
pocket.
Oh
really:
okay,
oh
yeah
yeah!
If
I,
if,
if
I
get
if
I've
got
different
schemas
for
different
versions
of
my
delta
table,
definitely
I'll
get
an
error.
Okay!
So
guys
what's
gonna
happen.
Folks,
what's
gonna
happen,
if
I
start
to
read
this
spark,
read
format
delta
loud.
A
A
What
geez
I
got
confused
by
starters
of
streaming!
Sorry,
okay,
anyway,
that
that's
the
beauty
of
you
know!
That's
the
beauty
of
live
coding
sessions
where
you
think
that
you
are
right
or
you
are
the
smartest
you
forget
about
this.
You
know
what
what
smarter
people
than
you
say
be
careful.
There
are
people
smarter
than
you.
You
just
neglect
this
fact
and
now
yeah.
It
happens
like
this.
Okay,.
A
A
Okay,
so
I'm
gonna
chuck
and
I'm
gonna
chuck
okay,
so
this
is
important
trigger,
which
I
did
already
and
now
I'm
reading
from
format
rate
well,
this
is
just
built
in
format.
It
could
be
kafka,
it
could
be
any
other
data
source.
You've
got
in
your
spark
application.
I'm
loading
data
from
this
format
in
a
streaming
fashion
in
similar
fashion,
so
I'm
creating
streaming
query
over
raid
data
source,
I'm
selecting
values
only
because
raid
gives
me
values
and
timestamps
and
I'm
changing
this
value
to
id,
and
then
I
write
this
stream
in
format
delta.
A
A
A
Okay,
anyway,
you
probably
did
not
notice,
and
you
are
not
going
to
ask
questions
about
this
because
well,
I
could
not
understand.
I
could
not
explain
it
so,
okay.
So
what
what
we
see?
Now
we
see
two
streaming
queries.
One
is
select
star
from
delta
table,
which
is
here.
Another
is
insert
into
okay.
So
we
see
that
yeah.
A
A
They
are
similar
in
nature
of
in
feature
in
features
to
delta
lake,
but
since
I've
got
no
experience
with
the
with
this
to
others,
I
cannot
claim
whether
they
are
better
or
worse
in
this
or
that
okay,
I
only
know
delta
lake
and
and
can
help
you
with
delta
lag.
But
again
I
heard
great
stories
about
hoodie
and
iceberg,
especially
that
there
are
some
index
based
optimizations,
something
that
that
you
may
find
in
commercial
version
of
the
lake,
something
that's
not
available
in
delta
lake
oss,
okay,
so
yeah!
So
it's
okay!
A
A
Nothing
will
ever
be
added
again,
so
we'll
get
to
this
delta
source
checking
whether
there
are
new
offsets
available
and
whether
there
is
something
to
be
pulled
from
delta
table.
Okay,
now
before
we
switch
to
some
other
topics,
let
me
do
this:
let's
review
tmp
delta
0
1
and
see
all
the
files
see.
There
are
plenty
of
files.
A
Some
are
for
version,
zero.
Some
are
for
version
one.
Now
it's
tricky
to
get
all
these
files
assigned
to
appropriate
versions,
so
write
versions,
that's
why
we
have
this
transaction
log
with
all
these
files,
json
files
every
now
and
then
you've
got
this
ever
10
files.
You've
got
this
parquet
file
with
checkpoint
information
based
on
these
json
files,
10
previous
json
files.
A
This
is
something
I'm
not
sure
what,
where
what
is
this
version
20
size,
how
many
files
we
have
20,
because
this
is
to
mark
the
last
checkpoint
available
just
to
make
looking
up.
You
know
checkmate
files
faster.
So
if
you
need
to
recreate
version
28,
for
example,
rather
than
reading
28
json
files,
you
know
that
you
query
last
checkpoint,
you
know
that
version
20
was
last
checkpointed.
A
You
read
this
file,
plus
all
these
files,
and
now
you've
got
version
28
of
your
delta
table.
That's
basically
it
okay.
The
reason
why
it's
so
complex
or
seemingly
complex
is
that
it
may
happen
in
the
cloud
on
hdfs
like
file
system,
which
is
inherently
slow
or
slow
by
design.
It's
not
supposed
to
be
fast.
It's
supposed
to
be
fault,
tolerant,
handling,
large
file,
large
number
of
files,
a
large
number
of
large
files,
and
so
so
it
may
be
slow.
It
might
not
necessarily
be
consistent.
A
A
A
A
What
what
is
was
supposed
to
do
kafka
kafka?
It
will
prohibit
reading
from
pocket
when
it's
delta
now
look
at
this.
It
was
okay
right,
but
it's
for
data
breaks.
Maybe
I
don't
know.
Okay
I'll
say
that
about
the
default
behaviors,
which
were
on
error,
important,
informal
detected
transaction
log
for
that
link
was
found.
If
you
run
yeah
false,
then
you
can
okay
yeah,
but
I
did
this
set
false
format
check,
maybe
but
delta.
A
This
is
only
for
delta,
so
okay
yeah
possible
that
it's
possible
and
possible,
but
it's
possible
yeah,
okay,
and
so
that's
it
for
now
and
let's
go
back
to
our
slides.
So
let
me
where
are
my
slides
here?
Oh
it's
here,
q,
a
if
you
insert
a
duplicate
and
duplicate
of
a
record
or
both
versions,
duplicate,
one
and
duplicate
to
start
in
delta,
table
e
or
just
one
if
you
insert
unless
you
do
upset
meaning.
If
you
find
this
record
and
then
you
update
it.
A
If
you
insert
the
same
record
every
day
for
a
year,
do
you
store
all
instances
of
a
record
or
just
one?
Well,
it's
basically
a
follow-up
to
your
early
question
right,
I'd,
say:
there'll
be
365
instances
of
the
record
and,
as
a
matter
of
fact,
we
can
just
give
it
a
shot.
So
I've
never
done
it
before.
A
A
Yeah,
it
was
only
possible
when
I
started
when
I
had
you
know
empty
directory,
so
append
override.
As
a
matter
of
fact,
I
can
only
append
right.
I've
always
been
confused
by
what's
possible.
There
are
safe
modes
like
overwrite,
append,
append
and
error
if
exist,
and
something
something
more
something
one
more,
if
I'm
not
mistaken,
but
then
yeah
append
well
by
definition
appendix
append.
So
now
you
know
what's
going
to
happen,
okay,
so
we've
got
three
new
records
and
then
let
me
do
this
again.
A
B
A
B
A
A
Given
this,
you
know
challenging
the
times,
I'm
hoping
that
you
all
are
you
know
holding
up
well
so
delta,
like
zero,
eight
zero
released,
let's
have
a
look
what's
what's
available,
what
they,
what
it
added
and
that's
how
I
marked
my
to
check
the
links
in
my
slides,
so
I
don't
have
to
remember
you
know
where
I
should
click
and
when
and
so
yeah
oops?
No,
no!
No!
No!
No!
No!
Not
yet
not
yet
okay.
So
what
are
the
features?
They
added?
A
Okay,
something
that
I
haven't
checked
yet,
and
I've
got
to
really
know
how
to
say
experience
how
important
it
is:
unlimited,
matched
and
unmatched,
and
there
there
are
some
optimizations
improvements
in
merge
operations
in
match
operation,
so
it
supports
schema,
schema
evolution,
nice
nested,
columns,
yeah.
Some
sounds
very
nice
to
have
match
into
update
nice
too.
A
It
was
to
start
streaming
a
table
from
a
specific
version.
So
now,
with
this
starting
timestamp
or
starting
version,
you
can
just
say
hey.
I
need
to
read
this
table
using
my
streaming
query
from
version
0
for
well.
0
is
default,
but
from
version
for
example,
5,
fifth
or
latest
they
are,
they
are
at
this
latest
mnemonic
or
alias.
So
you
can
just
skip
everything.
That's
available
and
well
start
as
if
you
started
from
scratch,
yet
well,
not
from
square
performance
which
would
be
from
zero
so
well
from
from
latest.
A
Well,
yeah
sounds
very
important
too.
Yet
I
haven't
found
time
to
to
dig
into
it.
Ability
to
perform
parallel
deletes
with
vacuum,
so
you
can
just
clean
up
files
that
are
no
longer
available,
no
longer
needed
and
something
that
may
ease
your
life
as
scala
developer.
So
after
you
import
this
implicit
object.
Well,
you
get
this
nice
extension,
so
you
can
just
spark
with
delta
rather
than
spark
read
dot
format,
delta,
okay,
so
yeah
nice,
nice
addition
as
if
I
was
not
myself
well.
A
I
just
found
this
quite
interesting
that
I'm
not
at
in
credits
yet
to
be
honest,
no
idea
what
how
I
contributed
to
this
release,
but
anyway,
thank
you
for
including
my
name
yeah.
As
I
mentioned
already,
delta
lag080
release
requires
spark
301,
it's
included
in
the
pom,
so
I
so
it's
a
spark
version
so
301.
A
A
If
at
all
no
failures
yet
except
there
was
one
error
message
which,
by
the
way
I
haven't
seen,
you
know
when
I
started
our
conversation,
it
was
about
something
about
antaler
parser,
something
that
was
something
about
me
using
spark
311
release
candidate
3,
which
probably
comes
with
latest
version
of
antalya
pasa
for
sql,
while
delta
lake
expects
something
lower,
and
there
was
some.
You
know
warning
from
antler
that
these
you
know
the
files
or
files
compiled
for
there
are
files
compiled
for
different
versions,
something
like
this.
A
A
Postgres
was
mentioned
now.
You've
got
it
available
in
spark
2
thanks
to
delta,
like
okay.
Beside
this
sql
known
statements
like
create
alt,
alter,
insert
delete,
update
merging
two.
There
are
some
delta
like
specific
sql
commands
among
them,
are
convert
delta,
which
is
to
convert
parquet
table
to
a
delta
table.
A
Okay,
these
are
extension
sql
commands
that
you
may
only
find
in
delta
lake
in
they
are
not
available
in
other
sql
environments.
Okay,
so
demo,
two
we've
got
half
an
hour.
If
I'm
not
mistaken.
Still
so
in
this
demo,
you
will
learn
how
to
use
these
sql
ddls
and
dmls
with
delta
lake
like
create
table,
describe
table,
describe
extended,
insert
into
describe
detail
and
describe
history,
and
you
know
what
every
time
when
I'm
touching
sql
itself
as
the
language,
I'm
a
little
bit
uneasy
with
what's
coming
because
I'm
not
sql
heavy
user.
A
A
A
But
I
know
how
important
it
is
that
delta
lake
and
spark
sql
support
both
support,
sql,
the
the
language,
structured,
query,
language
or
sql,
as
some
say,
yeah,
it's
it's
important
because
it
just
opens
the
door
the
doors
to
to
this
big
data,
distributed
computations
for
people
who
are
sql
heavy
or
savvy,
rather
than
you
know,
being
programmer
python,
scala
or
java.
Okay
same
with
python.
It's
it's
it's
important
to
to.
Let
them
you
know,
use
spark
for
for
something
that
they
thought
is
possible.
Has.
A
Possible
with,
for
example,
pandas
or
numpy,
okay,
so,
let's
dive
in
to
in
in
sql
support
in
delta
lake
okey
doke.
So
let
me
switch
and
again
this
is
streaming.
Query
is
still
up
and
running,
so
I
have
not
stopped
it
yet.
So
all
the
changes
to
my
to
my
to
our
table
are
going
to
be
reflected
by
this
assuming
query.
A
A
So
that's
where
antela
comes
on
stage
and
it's
used
so
yeah.
But
after
this
look
at
this,
you
will
see
no
warning.
So
this
is
just
for
the
very
first
time
and
that's
it
so
anyway,
so
that
that
might
be
the
reason
why
you
should
not
use
spark
311
with
delta
lake
zero
eight
and
you
should
expect
zero,
nine
zoom
or
zero
eight
one
just
to
clear
things
up:
okay,
so
no
tables!
Okay!
Okay!
A
A
There
is
this
configuration
option
so,
rather
than
repeating
this
using
delta
using
delta
or
just
not
even
telling
people
that
you
change
the
default
format,
they
would
probably
notice
nothing
different
if
you
switch
this
default
format
from
parquet
to
delta,
except
that
you
are
adding
something
that
they
may
eventually
like
and
awards.
You
eventually
too
so
end
with
this
comment.
Okay,
so
create
brand
new
table,
and
now,
when
you
do
show
tables
and
yeah
show
tables,
you
see
this
table
created.
A
Okay
and
what's
in
this
spark,
catalog
well,
you've
got
this
too,
but
the
difference
is
that
you
see
this
comment
that
was
added
by
using
this
comment.
I'm
not
sure
if
you
can
show
tables
to
the
point
to
get
this.
No
idea.
Okay,
if
you
know,
let
me
know
okay,
so
now
what
I'm
gonna
do
is
I'm
gonna
register
a
new
table,
but
not
by
using
sql
language,
but
by
saving
us
table
using
scala
api
for
spark
sql,
so
spark
range
basic
stuff.
I'm
writing.
A
A
A
A
B
A
You
know
familiar
with
hype
metastar
or
hive
metastar
like
things
when
I
started
exploring,
you
know,
spark
and
delta
lake
hive
and
and
mapreduce
were
all
in
at
full
swing
and
people
have
been
were
using
these
tools
very
heavily.
I
haven't
I
had
not
at
the
time
and
since
then
I
had
no
time
for
them
so
well
yeah.
So
I
built
all
my
understanding
about
this
hive
metastore,
like
things
by
looking
at
spark,
sql
and
spark
sql
like
solutions,
including
delta.
Like
so
I
apologize.
A
A
A
A
Delta
and
then
load
and
then
blah
blah
blah,
you
know
tmp
whatever.
Let
me
make
sure
that
it's
not
going
to
be
executed
so
yeah.
First
of
all,
we've
got
no
idea
what
where
this
d02
is
stored.
Actually
so,
and
it
does
not
really,
it
should
not
really
bother
us.
Okay,
so
well
that
that
that's
kind
of
like
a
alias
for
our
location,
which
I
found
quite
interesting
and
quite
useful
otherwise.
Well,
so
you
basically
have
no
idea
what
to
specify
as
load
to
touch
this
d02
table.
A
A
A
A
A
A
And
now
20c
well
yeah.
I
know
that
I
could
just
collect
and
get
this
126
to
be
automatically
added
here,
but
I
meant
to
do
this
more
interactively.
Okay,
so
I
got
this
126
and
I
remove
everything
every
record,
I'm
deleting
the
records
with
id
less
than
126,
leaving
just
one
record
the
one
with
id
126..
A
A
Yep
done
no
changes
here.
I
think
no,
it
was
not
with
delete.
If
I'm
not
mistaken,
it
was
when
schema
changed,
but
the
the
very
first
time
I
tried
this.
I
I
hoped
in
a
sense
that
I
I
expected
the
streaming
query
fail
because
well
this
information
in
this
dev
table
changed.
Since
this
streaming,
query
has
started
had
started
yet.
A
A
More
details
about
this
table,
we
just
deleted
126
records
from
okay.
So
first
of
all,
this
is
described
detail
about
this
table.
These
are
all
the
information
that
you
may
ever
get
from
delta
lake
and
spark
sql
about
this
delta
table.
Okay,
so
what's
the
format
what's
the
id?
This
is
the
id
that
we've
we
saw
in
the
logs
a
name
you
can
give
this
table
name
if
I'm
not
mistaken.
A
A
A
This
is
streaming
update,
streaming
update,
so
operations
have
names
and
there
are
some
operation
parameters
that
you
you
may
just
query,
and
for
for
some
some
monitoring
purposes,
some
other
information
user
metadata.
You
can
just
specify
user
metadata
using
user
metadata
option,
so
you
can
just
mark
some
transactions
special.
So
you
can
just
query
history
by
looking
up
entries
with
user
metadata
of
given
shape
or
given
you
know
value,
it
might
be
much
beneficial
for
your
reporting
purposes.
A
Okay
and
all
this
is
just
basically
another
view,
another
perspective
at
json
files.
This
is
exactly
what
you
may
have
figured
yourself
by
looking
at
json
files
directly,
okay,
good,
so
we've
got
seven
minutes
before
we
finish
and
that's
basically
it
as
far
as
this
demo
is
concerned.
So
let
me
see
what's
coming.
Aha,
ladies
and
gentlemen,
I'm
glad
that
this
is
exactly
the
the
moment.
Oh,
this
is
the
slide,
because
you
know
I
was
worried
that
time
travelling
might
not
be
covered,
which
was
not
good.
Okay,.
B
A
A
Scala
java
and
python
apis
all
support
this
version
as
of
option
as
well
as
timestamp
as
of
option,
so
you
just
specify
either
number
that
denotes
the
version
you
want
to
access
or
by
timestamp.
Okay.
Unfortunately,
no
support
for
these
options
in
sql
puresql,
it's
available
in
databricks,
sorry
to
say
it
so
hope
you
don't
mind
me
blaming
databricks,
not
open
sourcing.
A
This
very
cool
feature,
as
I
said
during
my
previous
meetup,
I
wish
databricks
focus
more
on
integrating
delta
lake,
with
the
platform
with
red
ash,
too,
and
and
making
sure
the
data
bricks
platform
is
stronger,
better,
not
by
stripping
down
a
delta
lake
but
by
you
know,
focusing
on
something
else,
but
hey
that.
That's
that's
how
it
is
so
no
z
index,
no
sql
support
for
for
version
yourself
or
timestamp
ourselves.
Sorry,
what
we
do
have
available
in
open
source
delta
lake
is
table
identifiers
path
with
time
travel.
A
A
Oh,
I
might
be
mistaken
if
I'm,
if
I,
if
it
is
possible
to
access
delta
like
tables
using
sql
with
this
time,
travel
we'll
see
what's
new
in
zero,
eight
is
that
you
can
start
streaming
a
table
from
a
specific
version,
so
you've
got
two
new
options,
starting
version
starting
timestamp,
and
so
so
now
you're,
basically
specifying
these
options,
and
you
can
just
start
from
from
the
table.
Why
did
this?
Why
did
they
not
leverage
version
ass
off
for
this
starting
version?
A
Okay,
I
have
to
ask
them
so
yeah
that
that's
definitely
possible.
So
let's
have
a
look.
We've
got
three
minutes
before
we
finish,
so
let
me
show
you
very
quickly
how
to
use
versions.
So
we
know
that
we've
got
this
sql
api
to
show
versions
of
our
delta
table,
but
we
can
also
do
something
like
this
io
delta
and
tables
and
delta
dt
yep.
A
Did
you
know
that
you
can
autocomplete
in
spark
show
by
dt
just
by
this
titles?
Now
it's
not
gonna
work.
Okay.
Let
me
check
again
now,
okay,
so
just
for
the
very
first
time
it
works,
then
it
stops
working
so
delta
table.
And
now
you
do
something
like
this.
You
access
this
delta
table
using
this
delta
table
programmatic
api
and
you
specify
that
you
are
going
to
access
delta
table
by
path
which
is
tmp
delta,
0
1.
It's
also
possible
for
python,
but
I
just
started
to
apply
them.
A
So
I'm
not
even
going
to
show
you
this
in
the
in
the
other
language
and
dt.
History
will
give
you
history
as
a
data
frame,
so
you
can
just
show
truncate
false
and
you
will
get
all
the
goodies
that
you
have
seen
in
sql
or
using
sql
right.
So
you've
got
this
delete.
Being
the
last
operation
nice,
so
now
let's
describe
history,
so
this
was
the
one
one
way
there
is
describe
history
delta.
This
way
it
gives
us
the
same
information.
So
you've
got
this
and
this
yep
both
work
same
okay.
A
So
now
let
me
time
travel
using
version
of
option
using
spark
sql.
So
this
is
batch
query.
So
what
I'm
gonna
do
is
yeah.
I
just
copied
it.
I'm
gonna
read
version
0
of
my
table,
even
though
I'm
it's
around
the
30th
version,
and
now
I'm
going
to
access
zeroth
version.
Is
it
possible
yeah?
It
is
so
two
records
only
what
if
I
specify
version
five,
for
example
or
fifth
or
maybe
you've
got
something
different
if
we
want
to
access
the
latest
version.
Well,
we
just
skip
this
and
we're
done
right
right.
A
So,
as
you
can
see,
this
is
the
beauty
of
data
lake,
which
gives
you
access
to
all
those
all
the
versions
that
are
available
in
transaction
log.
So
unless
you
vacuum
some
versions,
so
they
are
no
longer
available,
you
can
just
you
know,
time,
travel
back
and
forth
between
versions
and
just
just
compare
them
for
various
reasons.
Okay,
you
may
also
do
something
like
yeah
calculating
maximum
inversion
by
querying
this
desk
history
delta
table.
But
hey
you
don't
need
to
calculate
it
because
by
default
you
are
accessing
the
latest
version.
A
A
So
maybe
I'm
I'm
wrong,
saying
no
support
for
sql,
because
that's
exactly
what
I
meant
by
saying:
no
support
for
sqr.
Sorry
guys.
I
need
to
change
the
slides
yeah.
That's
that's
me
so
yep,
so
that
might
not
be
true,
no
more
okay
and
that's
pretty
much
it
guys,
folks
how
what
is
this?
Okay
anyway?
I
I
need
to
find
a
way
how
to
just
just
wait
about
just
just
cross
it
out.
Okay,
so
let
me
switch
to
this
and
finish
our
presentation.
A
A
You
can
find
more
interesting
information
than
what
I
just
covered
might
not
necessarily
be
that
low
level.
Yet
they
are
very
entertaining
too,
there
are
als.
There
are
also
competitors
covered,
hoodie
and
iceberg.
It's
written
the
book
and
the
chapter
is
written
by
jules
brooke
td,
which
is
the
the
main
author,
or
one
of
the
authors,
creators
of
delta
lake
and
spark
streaming
and
sparta
streaming
and
the
last
but
not
least,
the
only
danny
lee
okay,
my
mentor
these
days.
Thank
you
danny
for
sticking
to
all
my
questions
and
being
so
patient.
A
A
Delta
lake
properly
because
it's
basically
describes
motivation,
ideas
and
design
of
delta
lake
and
it's
written
by
by
the
the
brains
who
created
delta
lake,
michael
td
and
and
others
yeah,
okay
and
it's
free
of
charge.
So
basically
it
costs
you
just
your
time
and
yeah
the
last
button.
Please
there
is
this
book
free
of
charge
of
mine
in
terms
of
the
lake
I
keep.
I
I
try
to
keep
it
up
to
date.