►
Description
Online Tech Talk hosted by Denny Lee, Developer Advocate @ Databricks with Burak Yavuz, Software Engineer @ Databricks
Link to Notebook: https://github.com/dennyglee/databricks/blob/master/notebooks/Users/denny.lee%40databricks.com/Delta%20Lake/Diving%20Into%20Delta%20Lake:%20Unpacking%20The%20Transaction%20Log.py
The transaction log is key to understanding Delta Lake because it is the common thread that runs through many of its most important features, including ACID transactions, scalable metadata handling, time travel, and more. In this session, we’ll explore what the Delta Lake transaction log is, how it works at the file level, and how it offers an elegant solution to the problem of multiple concurrent reads and writes.
In this tech talk you will learn about:
- What is the Delta Lake Transaction Log
- What is the transaction log used for?
- How does the transaction log work?
- Reviewing the Delta Lake transaction log at the file level
- Dealing with multiple concurrent reads and writes
- How the Delta Lake transaction log solves other use cases including Time Travel and Data Lineage and Debugging
See full Diving Into Delta Lake tutorial series here:
https://databricks.com/diving-into-delta-lake-talks Databricks is proud to announce that Gartner has named us a Leader in both the 2021 Magic Quadrant for Cloud Database Management Systems and the 2021 Magic Quadrant for Data Science and Machine Learning Platforms. Download the reports here. https://databricks.com/databricks-named-leader-by-gartner
A
All
right
so
hello,
everyone
today,
we'd
love
to
talk
to
you
about
Delta,
Lake
and
one
of
I
would
say,
like
the
key
component
of
Delta
Lake,
its
transaction
log,
we're
going
to
go
into
the
details
of
what's
in
the
transaction
log
and
how
it
works.
So
before
we
start
I
just
want
to
introduce
ourselves
who
are
we
kind
of
like
a
philosopher
philosophical
question,
but
I
was
off
for
engineering
at
data
brakes
I'm
a
spark
committer
and
I've
been
working
on
Delta
Lake
for
about
four
years
now
we
open
sourced
it
last
year
April.
B
A
All
right
so
today,
the
way
we
want
to
introduce
Delta's
transaction
log
to
you
is
first
we'll
go
into
the
details
of
the
transaction
log.
What's
in
the
commit,
how
does
optimistic,
concurrency
control
work?
And
then
you
know
we're
going
to
talk
about
a
cool
feature
that
you
know
having
such
a
you
know,
multi
version
transaction
lock
provides
you,
which
is
time,
travel
and
we'll
try
to
show
it
all
in
action
through
a
demo,
but
before
we
start
I
just
want
to.
A
You
know,
like
I,
believe
everyone
knows
like
how
what
Delta
Lake
is,
but
you
know
in
a
very
short
way
to
explain
it
before
we
go
into
the
details
like
Delta
Lake
is
a
file
protocol
built
on
this
transaction
log
that
gives
you
transactionality
as
well
as
schema
enforcement's,
as
well
as
automated
file
management
on
top
of
files
that
you
store
in
your
data
leak.
So
the
transaction
log
is
the
key
piece
here
and
to
show
you
how
it
works
when
you
create
a
Delta
law
Delta
table.
This
is
what
it
looks
like
on
disk.
A
Alright.
So
let's
say
that
you
have
a
table
at
a
directory
called
my
table.
You
have
the
transaction
log
directory,
which
is
called
underscore
delta,
underscore
log
within
the
root
of
the
table.
That's
how
we
identify
that
a
delta
t.
You
know
this
table
is
a
delta
table
through
the
existence
of
this
directory.
A
A
Then
optionally,
if
your
table,
you
know,
there's
the
concept
of
partitioning
where
you
coarse
grain
index
your
data
through
some
common
values
of
your
data
set,
for
example.
Here
with
date,
we
could
have
partitioned
directories
where
your
files
may
live
in,
and
then
you
also
observe
your
files
within
these
partition
directories
or
at
the
root
of
your
table.
If
your
tables
not
partition
and
Delta,
underneath
the
hood
uses
open
source
parquet,
there's
nothing
special
about
these
parquet
files.
A
A
A
We
have
the
remove
file
actions,
so
one
of
the
key
pieces
of
Delta
is
that
it
provides
asset
transactions
and
one
of
the
key
things
here
is
with
asset
transactions.
You
need
isolation
and
what
that
means
is,
if
you're
running
a
query
and
some
other
operation
is
mutating
your
table,
the
query
should
run
on
an
isolated
snapshot
of
your
data
and
for
that
reason,
when
we're
deleting
records
or
when
we're
deleting
files
from
your
table,
we
can't
actually
physically
delete
them
immediately.
We
just
logically
delete
them
with
this
remove
file
action.
A
Then
we
have
a
set
transaction
action.
This
is
used
for
structured
streaming
whenever
you're
committing
files
to
your
table
through
structure
streaming,
we
add
new
set
transaction
actions
and
the
set
transaction
action
just
contains
the
ID
of
the
stream.
The
ID
of
your
stream
is
exists
in
the
checkpoint
location
of
your
stream,
and
it
also
contains
the
epoch,
ID
or
the
batch
ID
of
how
long,
how
many,
how
many
batches
it
has
committed
for
you
know
that
stream,
specifically
so
by
just
using
this
set
transaction
action.
A
We
can
say
that
oh,
this
stream
has
committed
up
to
the
batch
35.
If
it's
trying
to
recommit
batch
34,
we
can
just
say:
oh,
we
already
have
that
data.
You
don't
need
to
commit
that.
Then
we
also
have
a
protocol
version
for
your
Delta
tables.
This
provides
kind
of
guarantees
that
you
know
if
you're,
using
an
old,
Delta,
Lake
client
when
trying
to
read
from
your
table.
This
ensures
that
your
old
clients
or
your
newer
clients
can
actually
deal
with
all
the
metadata
and
information
within
the
transaction
block
of
your
table.
A
So
this
is
kind
of
like
if
you've
ever
used.
Kafka
Kafka
also
has
like
client
versions
that
are
Protocol
versions.
So
this
is
the
exact
same
with
Delta.
This
is
just
to
provide.
You
know
backwards,
compatibility
as
well
as,
if
we're
ever
breaking
forwards,
compatibility
you
know
about
it,
then
we
also
have
some
metadata
about
each
commit,
and
this
is
contained
in
the
commits
info
action
and
basically,
once
you
get
to
replay
the
transaction
log,
you
know
add
and
subtract
all
these
actions
in
a
linear
manner.
A
A
A
One
thing
to
understand
is
the
transaction
log
for
a
delta
table
is
the
single
source
of
truth
for
your
Delta
table,
so
any
reader,
that's
reading
for
your
Delta
table
will
take
a
look
at
the
transaction
log.
First
therefore,
changes
to
the
table
our
stored,
as
ordered
atomic
units
called
commits.
So
here
we
have
serial
JSON
we
have
and
then
once
you
make
new
changes
to
your
table,
they
will
appear
as
one
dot
JSON
when
you're
querying
your
Delta
table.
This
one
JSON
contains
all
the
changes
since
version
zero
of
your
table.
A
So
we
can
see
here
that
you
know
in
the
first
version.
Let's
say
you
know:
two
files
were
added
one
dot
Park
and
to
adopt
arcade,
and
in
the
second
version
we
say
that
we
removed
one
dot.
Parkade
remove
two
dot
party.
We
did
some
compassion
process
to
add
back
three
dot
parking.
So
how
would
this
work
as
we
play
these
lists
of
actions
linearly
in
order?
The
removes
will
cancel
the
adds
and
we
will
end
up
with
the
final
state
of
the
table
being
there's
a
single
file
in
this
table
called
3
dot
parking.
A
Another
key
piece
here
that
you
know
guarantee
that
Delta
requires
is
mutual
exclusion,
so
we
need
to
agree
on
the
order
of
changes
even
when
there
are
multiple
writers-
and
this
provides
a
guarantee
and
database
is
called
serializability.
The
fact
that,
even
though
things
are
happening
concurrently,
you
could
play
them
as
if
they
happened
in
a
synchronous
ordered
manner.
So
we
have
zero
dot
JSON.
You
could
have
two
writers
trying
to
write
one
dot,
JSON
and
two
JSON,
so
you
have
user
one
that
reads:
zero.
A
You
have
user
2
that
reads
one
as
you're
trying
to
commit
user,
two
dot,
JSON
user,
two
wins
and
user.
One
takes
a
look
and
sees
that
Oh,
two
dot.
Json
is
already
there
yeah
due
to
this
requirement
of
mutual
exclusion.
Therefore,
it
will
have
to
say:
oh
this
commit
failed.
Let
me
try
to
commit
three
dot
JSON
instead,
so
how
does
that
work?
A
Within
Delta
within
databases?
There
are
two
modes
of
concurrency
where
you
can
have
pessimistic,
concurrency
or
optimistic
concurrency
within
Delta.
What
one
means
is
with
pessimistic.
You
think
that
there
are
always
going
to
be
transaction
conflicts,
so
in
order
to
provide
the
best
user
experience,
you
tell
you
lock
your
entire
table
as
you're
doing
your
transactions
within
Delta.
We
chose
optimistic
concurrency
because,
with
big
data
workloads,
you
tend
to
append
more
data
to
your
table
rather
than
having
like
multiple
riders
consistently
trying
to
update
the
same
records.
A
Updating
the
same
records
is
typically
rare,
at
least
concurrently.
So
the
way
that
optimistic,
concurrency
works
is
that
we
first
record
the
start
version.
Then
we
record
the
reads
and
writes
of
that
version,
and
then
we
try
to
attempt
to
commit,
and
if
someone
else
wins,
you
have
to
check
all
the
changes
that
have
happened
since
your
read
version
and
see
if
anything
changed
with
regards
to
what
you
read.
A
So
if,
in
the
cases
where
you
know,
if
there
are
two
appends
coming
in
at
the
exact
same
time
since
you
didn't
read
anything
you
know
what
would
happen
is
you
would
have
two
writers,
they
would
write
out
the
data
files.
First,
then
one
writer
will
make
the
commit
and
say
that
okay
I've
got
all
these
ads.
Then
you
have
your
second
writer,
which
already
wrote
out
all
the
data
files,
but
it
sees
oh
there's
already
a
commit
that
I
wanted
to
commit
at
so
commit
version.
A
2
has
already
made
it
in
I
need
to
check
what
I've
changed
in
commit
version
2.
So
it
reads
what
happened
in
commit
version.
2
sees
that,
oh
all
of
those
were
it
pens
I'm,
also
a
pure
append,
I,
don't
care.
So
it
just
says.
Let
me
just
come
in
version
0.3.
It
doesn't
have
to
rewrote
all
right
any
felt
it
doesn't
have
to
redo
the
entire
computation
again
it
just
automatically
commits
version
3
in
other
cases.
A
A
A
So
with
regards
to
you
know
how
Delta
scales
is
that
you
know,
since
we're
dealing
with
big
data,
we
have
a
ton
of
files.
We
could
have
tons
of
partitions
as
well.
How
do
we
scale
the
metadata
because
they're
just
stored
in
you
know
as
JSON
files?
So
what
we
do
is
we
use
part
for
scaling?
So
let's
say
you
have
all
these
actions
committed
through
multiple
versions
of
your
data.
A
We
could
read
all
these
JSON
files
as
a
spark
job
and
then
every
now
and
again,
every
10
commits
we
automatically
create
a
check
point
that
contains
the
entire
state
of
your
table
within
one
part.
A
table
within
one
part.
A
file
then
spark
can
read
this
parky
file
or
spark
can
read
these
JSON
files
play
the
actions
in
a
linear
order
and
see
what
the
latest
state
of
the
table
is.
So
this
is
much
better
than
going
into.
A
You
know
hitting
your
hive
metaphor
to
continuously
get
metadata,
because
eventually
you
know
the
hive
meta
store
will
have
you
know
thousands
of
tables.
You
know
tens
of
thousands
of
partitions
and
like
concurrently,
hitting
this
one
single.
You
know
database
to
answer
all
these
queries.
You
know
you'll
hit
bottlenecks,
whereas
with
spark
each
of
your
clusters
are
individual
or
I
mean
each
of
your
processes
are
individual
processes.
Each
of
your
tables
have
their
metadata
separately.
So
you
have
a
better
scaling
guarantees
my
used
spark
on
the
transaction
log.
A
A
So
whenever
you're
acquiring
your
Delta
table,
we
can
use
this
in
memory
representation
of
your
table,
which
is
you
know,
only
megabytes
of
data.
It's
very
tiny
to
answer
all
your
queries,
but
what
happens
when
you
get
new
data,
so
you
were
on
version
2.
What
you
do
is
you
start
listening
your
table
and
then
you
see
all
these
new
versions
pop
up.
So
let's
say
that
five
commits
have
been
made
since
you
last
queried
your
table,
so
the
latest
version
of
the
table
is
version
seven.
A
A
Then,
when
you're
at
seven-
let's
say
five
more
commits
made
it
in,
as
you
can
see
this
time,
we
have
a
checkpoint
file
that
happened.
You
know
at
the
tenth
commit
this
checkpoint
file
contains
all
your
changes.
I
mean
all
the
changes
since
version
10.
So
when
we
list
from
version
seven,
what
we're
going
to
see
is:
oh
there's
a
checkpoint
file
and
then
there
are
two
more
versions
after
that.
A
A
A
What's
the
latest
checkpoint,
we
can
see
version
10
cemani
at
having
a
checkpoint
play
the
one
commit
right
after
it
and
come
out
to
version
1071
while
you're
time
traveling
by
timestamp.
Then
the
case
happens
is
that
you
can't
just
call
get
snapshot
at
1071.
You
need
to
figure
out.
You
know
which
version
that
timestamp
corresponds
to
within
your
table,
to
to
explain
this
concept.
I'm
gonna
use
some
data
from
the
Middle
Ages.
A
A
So
we
take
a
look
at
all
your
commits
and
all
your
commit
time
stamps
come
from
storage
system
modification
time
stamps.
What
does
this
mean
if
you
use
HDFS
and
you
write
a
file
to
HDFS?
It
assigns
a
modification
timestamp
to
your
commit
file,
same
thing
with
Amazon
s3
same
thing
with
Azure
data
leak.
So
we
use
these
timestamps.
As
kind
of
you
know,
the
source
of
truth
of
when
these
commits
made
it
into
your
Delta
table,
but
the
problem
is,
these
are
distributed.
A
Systems
and
distributed
systems
can
definitely
have
clock
skew
so
in
this
example,
I've
kind
of
used
Turkish
history
to
demonstrate
this.
You
know
time
stamps
375
was
one
the
Turks
migrated
from
Central
Asia
into
Europe
and
Anatolia,
or
towards
the
West
instead
of
staying
in
Central
Asia,
and
that
was
the
beginning
of
the
Middle
Ages
1453
is
when
Istanbul
was
conquered
by
the
church
or
the
Ottomans
and
in
the
West.
A
So,
as
you
can
see,
there's
some
inconsistency
between
when
the
parliaments
opened
and
when
the
Republic
was.
You
know
when
the
Republic
came
to
be.
So.
How
do
we
deal
with
this
since
time?
Stamps
can't
be
out
of
order.
We
have
to
list
all
these
commits
and
we
adjust
these
time
stamps
by
adding
one
millisecond
to
any
time
stamp.
That's
out
of
order.
So,
as
you
can
see,
version
10
73
will
suddenly
have
1
millisecond
after
the
1923
version.
A
Then,
when
we're
trying
to
find
oh,
when
was
1492
within
these
lists
of
commits
we
play,
we
look
at
the
latest
version
that
has
the
closest
time
stamp
that
doesn't
exceed
the
given
time
stamp.
So,
as
you
can
see,
1453
happened
right
before
1492
and
we
can
therefore
choose
version
1071
to
actually
understand
which
version
that
we
have
to
load.
And
then
we
can
call
Delta
log,
get
snapshot
at
1071
to
replay
that
timestamp.
So.
A
Kind
of
limitations
with
time
travel
first
of
all,
it
requires
transaction
log
files
to
exist
with
Delta.
We
keep
by
default
30
days
worth
of
transaction
logs,
I
mean
30
days
worth
of
logs
in
changes
for
auditing
purposes,
and
these
can
be
modified
by
the
log
retention
duration
configuration
within
Delta.
A
It
requires
the
data
files
to
exist
because,
as
you
do
compactions,
you
logically
remove
old
files,
but
once
you
call
vacuum
on
your
table
to
delete
these
logically
deleted
files,
you
know
you
get.
You
lose
the
chance
or
opportunity
to
go
back
in
time
and
read
those
files
so
to
prevent
that
you
can
also
set
like
a
deleted
file
retention
duration,
equal
to
your
log
retention
duration,
to
maintain
full
compatibility
of
the
entire
history
that
you
can
time
travel
to.
A
The
thing
here
is
that
time,
traveling
in
order
of
months
and
years
is
infeasible,
and
the
reason
for
that
is,
your
storage
costs
will
just
explode.
So
if
you
want
to
be
able
to,
you
know,
go
back
in
time
to
see
how
a
record
what
what
a
record
was.
You
know
a
year
ago
or
two
years
ago
or
10
years
ago.
Even
then,
you
know
there.
A
You
have
to
change
your
architecture
in
such
a
way,
whereas,
like
you
can
store
your
data
in
SCD
type,
2
format,
for
example,
which
retains
the
entire
history
of
you
know
all
the
changes
to
each
of
your
records.
But
when
you
query
the
data,
it
always
shows
you
the
latest
value,
thanks
to
like
a
boolean
flag,
which
is
part
of
your
table
schema
so
do
not
use
sun
traveled.
A
For
you
know,
years
and
years
worth
of
you
know,
history
time,
travel
is
kind
of
you
know
the
way
we
built
it
initially
was
to
prevent
mistakes
from
happening,
because
we've
observed
a
lot
of
users,
first-time
users,
you
know
misusing
Delta
and
accidentally
deleting
all
their
data.
With
time
travel
you
could
very
easily.
You
know
store
restore
all
your
deleted
data
because
it
wasn't
actually
like
really
physically
deleted.
A
And
another
reason
is
that
computing
Delta's
state
won't
scale
when
you're
deleting
data.
If
you're
trying
to
you
know
contain
you
know
years
worth
of
history
and
the
reason
for
that
is
we
delete
checkpoint
files
a
little
more
aggressively.
Therefore,
you
might
end
up
with
cases
where
you
might
have
to
load
one
checkpoint
file
and
a
hundred
thousand
JSON
files.
So
if
you
do
need
history,
you
know
there
are
architectural.
B
B
B
So
this
notebook
is
a
just
to
give
you
some
context
for
everybody
here,
it's
actually
available
already,
but
we
will
share
out
this
specific
version
of
the
notebook
shortly.
Okay,
now
this
is
a
version
of
the
Delta
Lake
tutorial
from
spark
and
AI
summit
I'm
from
2019
in
Europe.
But
what
we're
doing
now
is
we're
gonna,
we've
done
a
modified
version
specifically
for
this
talk.
Okay,.
C
B
So,
as
you
save
diving
into
Delta,
Lake
unpack
in
the
transaction
log,
so
I've
already
started
some
of
the
running.
Basically,
what
I've
done
is
I've
downloaded
some
data
specifically,
which
is
specifically
some
loan
risk
data
that
we
had
already
generated.
Okay,
so
this
is
just
downloading
the
file.
It's
a
relatively
small
file
and
I've
cept.
B
If
you
in
case
you
want
to
run
this
on
data
books,
Community
Edition,
you
absolutely
can
so
we've
already
configured
it
as
such,
so
that
makes
it
easier,
and
the
first
thing
we're
doing
is
we're
creating
a
park
version
of
the
table.
So
in
other
words
we're
not
gonna
go
ahead
and
just
go
to
Delta
right
away.
We
just
want
to
start
with
part
K
to
give
you
some
context
behind
what
Barack
actually
was
was
talking
about
to
give
you
some
context
of
how
does
it
operate
when
you
just
go
with
Park
Hae
alone?
B
Okay
inside
created
this
Park
a
table,
I
defined
a
view,
loans
Parque
here
so
hopefully
you'll
be
able
to
see
it
I
make
sure
I'm
gonna
zoom
one
more
time,
so
you
can
see
it
better.
Okay,
so
loans,
Parque
and
you'll
see
the
schema
of
this
table.
This
table
has
four
columns:
loan,
ID,
funded
amount
paid
amount
and
the
state
alright.
So
how
many
records
do
you
have
we're
just
an
account
on
that
loans,
parque
and
what
you'll
see
is
fourteen
seven,
oh
five.
Alright!
B
So
that's
the
number
du
jour
first
today,
alright
and,
as
you
can
tell
I,
just
ran
it
so,
but
now
I'm
gonna
go
ahead
and
do
kickoff
that
structure
streaming.
That
Brock
had
referred
to
earlier
in
giving
the
context
of
the
owner
run,
structure
streaming
and
batch
at
the
same
time,
because
Delta
Lake
allows
you
to
do
that
because
it
has
asset
transactions
to
protect
everything.
So
let's
go
ahead
and
just
create
this
particular
function
and
let's
go
ahead
and
run
it
okay.
B
B
So
if
I
was
to
go
ahead
and
do
a
count
on
the
same
path
now
note
the
fact
that
I've
indicated
here
Park
a
path
for
Matt
Parque,
but
what
I'm
doing
basically
is
I'm
not
doing
a
count
on
the
stream
I'm
doing
count
on
the
batch.
So
in
fact,
I
can
go
ahead
and
do
that
which
is
pretty
nice.
Okay,
so
is
right
now
telling
me
160,
but
there's
a
couple
of
problems
here:
I'm
actually
gonna
address.
B
Two
of
the
first
thing
is
what
happens
to
I
at
when
I
add
try
to
add
a
second
street
okay,
so
the
exact
same
function,
we're
just
generating
different
sets
data
and
we're
gonna
write
to
the
same
location,
Parque
path?
Okay,
so
when
I
do
that,
basically
I'm
going
to
kick
off
the
stream?
Well,
you
notice
is
that
basically
over
time,
while
the
this
stream
there's
no
input
processing
rate,
there's
actually
not
writing
any
data.
The
backs
raishin
is
gonna,
go
up,
but
there's
actually
nothing
being
written
here.
B
Okay,
this
this
other
one
other
stream,
this
first
stream.
It's
actually
writing
still.
Okay,
at
you
know,
forty
one
point:
five
records
per
second
input
rate
is
five
records
per
second
okay,
but
nothing's
happening
here.
Okay,
so
it's
good
that
I've
got
data
going
in,
but
I
can't
put
a
second
stream
in
so
in
other
words,
if
I'm
building
these
data
pipelines
were
I'm
actually
trying
to
scale
and
process
multiple
threads
from
multiple
sources
or
a
partition
source
and
push
them
down
to
my
table.
B
I
can't
do
that,
okay,
so
the
good
news
is
that
there
is
data
going
in,
but
I
can
only
do
one
stream,
so
in
essence,
I've
locked
my
table.
Alright,
that's
problem.
Number
one
problem
number
two
is
wait,
didn't
I
just
say
there
was
fourteen
thousand
seven
hundred
five
rows
right.
So
right
now
from
the
last
grab
410
what
happened
to
all
my
data?
Okay?
Well,
if
I
look
at
the
schema
here,
what
you'll
see
here
right
away
is
that
the
problem
I
have
is
that
I
do
have
loan.
B
I
defunded
amount
paid
amount
majeure
state,
the
four
columns
that
I
originally
started
off
with,
but
I'm
actually
miss
it,
but
I've
actually
have
two
additional
columns,
time
stamp
and
value.
So
this
is
cool
in
terms
of
I
had
a
time
stamp
of
when
this
data
went
in,
but
it's
not
cool
because
I
just
basically
corrupted
my
table
for
all
intents
and
purposes
it
doesn't
I,
don't
have
the
actual
account,
but
I
want
it.
So
this
isn't
good.
B
Alright,
so
I'm
gonna
stop
all
the
streams
right
now,
but
basically,
what
it
comes
down
to
is
because
I'm
using
a
perk,
a
table
I
can't
actually
run
multiple
streams
against
my
table
at
the
same
time,
okay,
nor
can
I
once
there's
a
schema
change.
It
basically
in
essence
corrupts
the
data
that
I
have
okay,
so
both
are
pretty
not
fun,
are
options:
okay
and
common
enough
when
you're
working
with
in
any
data
engineering
pipelines?
Okay.
So
let's
go
ahead
and
do
this
now
with
Delta
Lake,
so
I'm
gonna
rerun
this
particular
portion
of
again.
B
This
is
just
recreating
the
views
except
I'm
cleaning
it
up
and
I'm
gonna
create
a
loans.
Delta,
okay,
and
so
when
I
run
this
again
you'll
see
the
same
14-7
five.
So
that's
a
good
starting
point.
Okay
and
again,
when
we
look
at
the
data,
there
are
the
four
columns
that
we're
working
with
okay,
so
perfect,
alright,
so
we're
back
in
action
or
using
a
Delta
table
instead
of
Park
a
table.
Okay,
but
then
can
we
do
the
things
that
brach
I
just
mentioned
all
right?
B
Well,
let's
go
ahead
and
start
talking
about
a
little
bit
about
schema
enforcement
by
the
way
as
I
talk
about
scheme
enforcement,
I'm
only
gonna
talk
about
it
a
little
bit.
In
fact,
Andreas
and
myself
will
have
a
session
next
week
where
we
dive
into
schema
enforcement
and
schema
evolution.
Okay,
so
we're
just
gonna
cover
a
little
bit
here,
just
to
give
you
some
context,
but
we're
gonna
dive
in
much
deeper
in
the
next
session.
So
this
is
a
little
call-out
for
ourselves.
B
Okay,
so
I've
created
the
view
now
note
that
I'm
actually
creating
a
view
called
loans,
Delta
Street,
okay
off
this
Delta
path,
location,
okay,
so
it's
one
file
system,
location
that
that
in
which
I'm
creating
a
view.
That's
for
streaming
purposes:
okay,
so
I'm
now
gonna
do
a
count
against
that
and
it's
basically
trying
to
process
the
data
okay.
So
it's
just
simply
trying
to
do
a
count
off
that
that
file
path,
that
Delta
path,
location
here.
So
now,
I'm
gonna
go
ahead
and
try
to
run
that
same
code
again.
B
Remember
it
tried
to
put
two
additional
columns.
The
reason
it
does
is
because,
by
definition,
when
you
run
a
stream,
it'll
actually
add
those
times
that
value
columns
automatically
aha,
but
there
is
now
schema
mismatch.
So
this
is
the
one
cool
one
of
the
cool
things
about
Delta
Lake
right
away
is
that
there
is
schema
enforcement
it'll
protect
the
table
right
from
the
get-go,
so
it
doesn't
allow
this
function
to
write
to
that
same
table
because
there's
a
schema
mismatch.
B
Basically,
the
original
table
schema
has
these
four
columns,
but
the
data
schema
that
we're
trying
to
write
to
has
six
columns,
so
you
can
go
ahead
and
basically,
if
you
were
to
change
it
or
you
to
turn
on
offshored
schema,
then
you're
good
to
go
alright,
alright,
but
let's
go
rewrite
the
function.
So
all
we
did
is
basically
add
this
particular
statement
here.
So
we
saying,
let's
only
write
the
four
columns
that
you
see
here
all
right.
Let's
go
run
this
down.
B
Okay
and
now
let's
go
ahead
and
write
this
again,
so
we're
gonna
do
the
exact
same
thing
as
we
did
before
we're
gonna
go
ahead
and
run
and
we're
gonna
run
this
particular
one
to
basically
write
to
that
Delta
table
just
the
same
process
like
we
had
saw
it
before.
So,
if
I
open
this
up,
I'll
see
the
input
processing
rate
kick
in
and
per
the
batch
duration,
and
if
I
give
it
a
couple
seconds
when
I
scroll
back
up,
you're
gonna
see
the
numbers
change.
B
B
All
right
right
here
at
you
know,
50
records
per
second
input
rate
and
106.7
records
per
second
for
the
processing
rate,
and
this
is
for
the
second
stream
and
the
first
stream
same
concept.
Okay,
so
both
streams
are
writing
to
this
table.
At
the
exact
same
time,
all
right,
why
ask
the
transactions
are
protecting
it
if
I
scroll
back
up
you'll
notice
that
the
counts
are
quickly
quickly,
updating
those
cells.
So
now
that
what
you
see
here
is
I've
got
two
streams
that
are
writing
to
the
same
file
system.
B
I've
got
one
stream,
that's
also
reading
from
that
same
file
system.
Okay,
that's
why
we're
2505,
if
I
scroll
down
and
run
a
batch
I'm
gonna
run
a
batch
at
the
exact
same
time
as
well.
So
now,
I
have
a
batch
count
all
going
at
the
same
time,
so
concurrently
to
rate
stream,
sorry
to
write
streams,
one
read
stream
and
one
read
batch.
Okay,
so
gives
you
that
flexibility
to
run
multiple
things
at
the
same
time
against.
B
B
What
you'll
notice
here
actually
is
that
remember
that
the
data
was
sitting
in
this
location,
loans,
Delta,
okay,
what
happens
is
that
there's
actually
a
underscore
delta
underscore
log,
okay
folder
as
well,
so
if
I
run
that
what
you'll
see
is
a
bunch
of
files
here,
okay,
so
here's
the
0th
file,
the
JSON,
alright,
that's
the
one
that
actually
initially
created
the
table
and
for
each
JSON
that's
been
added.
This
is
the
transaction
log
that
you
that
you're
calling
out
and
just
had
Braga
noted
in
when
he
had
mentioned
before
every
tenth
one.
B
We
go
ahead
and
actually
switch
over
to
our
parque
to
speed
things
up.
Okay
and
right
now
there
go
the
here's
the
second,
because
this
is
the
twentieth
file
and
so
right
now
we're
on
version
21
or
22,
while
we're
of
versions
of
the
history
for
our
Delta
table.
Okay,
so
you
can
you
don't
have
to
look
at
it
like
this?
B
Okay,
you
can
actually
look
at
it
in
a
much
nicer
fashion,
so
we're
gonna
go
ahead
and
actually
simply
run
the
Delta
table
history,
okay,
so
this
is
just
by
the
way
to
provide
some
some
context.
This
is
actually
all
the
open
source
code
we're.
We
happen
to
be
running
this
on
Delta
on
on
data
bricks,
but
you
can
actually
run
this
on
the
OSS
code
yourself,
okay,
and
so
here
are
the
22
different
versions
of
your
table.
Okay,
alright.
B
So
from
the
first
initial
write
to
all
the
streaming
updates
that
went
in
to
your
table:
okay,
all
22
versions,
just
like
the
file
system,
and
so
it
just
this
also
a
bracket
noted
previously.
If
I
was
to
look
at
the
file
all
inside
each
JSON.
Is
all
this
committee
formation
all
those
metadata
right,
the
time
step
they
used
right,
D?
Who
I
was
who
did
it?
What
was
the
actual
operation,
the
parameters,
the
append
by
query,
ID
isolation,
levels,
the
transaction
yada,
yada
yada?
B
All
this
information
per
each
file
right
is
all
here,
so
it
gives
you
each
batch
each
transaction,
you
understand,
exact,
what's
being
impacted,
what's
being
upended,
what's
being
modified,
what's
being
changed?
Okay,
so
that's!
What's
inside
these
JSON
files,
all
right
and
then
we're
gonna
cover
this
a
little
bit
more
in
terms
of
in
some
of
the
future
sessions
when
we
do
especially
the
DML
operation,
we're
doing
in
certain
updates,
but
just
to
give
you
a
some
quick
context.
B
Jess
as
Brooke
had
noted,
I
can
now
with
time
travel
go
ahead
and
say:
okay,
give
me
the
version
you
can
do
time
stamp
too,
but
for
the
purpose
of
this
demo,
I'm
just
gonna
do
version,
and
so
what's
vo1
the
data
frame
here,
I'm
just
gonna,
be
saying:
okay
format,
Delta
give
me
the
table.
Okay
from
that
delta
path,
version,
1
versus
version
11
versus
the
current
version,
okay,
whatever
that
version
is
in
this
case,
since
we
did,
this
live,
is
version
22
and
then,
as
you
see
here,
here's
the
counts.
B
So
initially
the
table
was
14
705,
all
right
by
version
11,
there's
18,500
five
rows
inside
it
and
for
the
current
version
they're,
currently
24,000.
So
that's
really
cool
but
just
said
is
broken.
None
it.
How
can
I
actually
have
all
the
data
still
there
like?
After
all,
you
just
can't
stay
inserted.
What
why
can
I
know?
What
did
it
happen
on
version
1
versus
version
11
versus
the
current
version?
Well,
it's
because
there's
a
lot
of
files,
so
I'm,
actually
just
gonna.
Do
a
quick
query.
You're
gonna
see
a
lot
of
files
here.
B
Ok,
so
here's
all
these
little
files,
all
these
little
small
park'
files
that
have
been
added
right
inside
that
directory.
Ok,
so
it
starts
off
with
the
sorry
the
Delta
log
like
I'd
mentioned
before,
but
here's
all
those
files,
ok,
they're,
all
pretty
small,
as
you
can
tell,
and
I'm
actually
going
to
scroll
all
the
way.
The
bottom
and
there's
like
here's,
the
initial
file
that
we
had
created
right
for
the
initial
table
creation,
but
otherwise
of
that
lots
of
small
little
files.
So,
as
you
can
probably
guess,
this
is
gonna
be
slow
right.
B
If
you
keep
on
adding
more
and
more
files
into
this,
this
will
become
a
very
slow
process,
trying
to
read
that
data
become
slow.
It's
a
management
headache
all
that
fun
stuff.
Ok!
So
how
do
we
avoid
that
problem?
Well,
we're
gonna
talk
about
compaction
of
files
and
also
vacuum
me
all
right.
So,
first
things.
First,
let's
go
ahead
and
compact
the
files
alright.
B
What's
repartition
the
number
of
files
I
want
for
I'm
doing
this,
just
because
in
this
particular
case,
a
good
rule
of
thumb
or
good
starting
point
is
that
I
have
four
worker
nodes,
so
yeah
I'm,
just
gonna
go
before
right
and
then
I
have
to
specify
where
the
data
change
is
false
by
default.
The
data
change
is
true.
Okay,
so
when
it's
false
it
says:
okay,
now
I'm
gonna
go
ahead
and
actually
make
changes.
I'm
not
really
gonna
change
assume.
B
There
are
no
changes
to
the
data,
so
I
can
actually
cleanly
read
and
actually
run
the
operations
and
it
will
change
the
data.
Basically,
alright,
so
I'm
gonna
keep
it
as
of
the
format.
Delta
I'm,
gonna,
overwrite
it
in
Kate
and
then
basically
save
it
to
the
Delta
path.
Okay,
so
I'm
gonna
run
this
now
so
what
happens
here
is
now
I
am
quickly
going
to
create
four
new
files.
B
Alright
little
bit
four
point
three
two
seconds,
so
that's
pretty
sweet,
except
if
I
was
to
go
ahead
and
look
at
this
all
over
again,
I
still
see
a
lot
of
files.
Okay,
all
right!
Why
do
I
still
see
a
lot
of
files,
but
I
see
a
lot
of
false
because
and
in
fact,
if
I
should
just
do
a
count,
oh
there
there
329
okay,
so,
but
why
are
they
so
many
files?
Well,
the
reason
why
there's
so
many
files
is
because
of
for
snapshot,
isolation
purposes.
B
If
there
is
somebody
trying
to
query
the
data
at
the
exact
same
time
as
I
was
running
that
operation,
they
can
still
read
the
old
files
they're.
Still
there
before
they
go
ahead
and
try
to
read
the
new
files,
okay,
so
that
makes
things
a
lot
better
from
a
concurrency
perspective,
where,
basically,
you
can
have
multiple
people
querying
at
the
same
time,
they
can
still
go
ahead
and
actually
look
at
the
older
files,
while
you're
making
you're
performing
your
operations
alright.
So
but
it's
okay!
Let
me
go
ahead
and
just
simply
clean
up
that
data.
B
Now
the
due
note
in
production,
this
is
an
anti-pattern.
Okay,
do
not
do
this
in
production
by
default,
when
I
run
a
vacuum.
What
I'm
doing
is
without
without
specifying
a
partition
the
horse
equals
zero.
What
I'm
doing
is
I'm
saying
go
ahead
and
by
default
remove
any
older
files
that
are
seven
days
or
older.
Okay,
you
can
change
that
number.
B
B
Okay,
all
the
log
is
still
there
all
the
transactions
that
you
just
did
are
still
recorded
so
from
a
gdpr
or
you
know,
CCPA
or
GRC
compliance
perspective.
You
still
have
that
information
on
what
happened
to
the
table,
but
the
underlying
data
no
longer
exists,
as
you
can
tell
because
I
just
vacuumed
it
out.
Okay,
so
that
should
cover
it
for
the
demo
we
do
actually,
like
I
said
we
have
actually
these
demos
available.
We're
gonna
be
sending
this
demo
out.
B
But
if
you
don't
can't
wait
the
five
hours
for
us
to
go
ahead
and
email
this
out
to
you
or
updated
in
YouTube,
it's
okay,
you
can
go
to
the
SPARC
summit,
Europe
demo,
right
that
we
had
in
the
in
the
Delta
repo,
but
otherwise
that
we
will
be
sending
it
out
inside
this
notebook.
We
also
call
it
that
you
can
join
Delta
Lake
on
github.
Obviously,
that's
where
we
put
a
lot
of
our
stuff,
also
the
Delta
Lakes
slack
Channel.
B
A
You
know
in
order
to
maintain
this
mutual
exclusion
property
within
its
transaction
lock,
has
specialized
implementations
of
its
lock
store,
depending
on
which
you
know,
storage
system
you're
on,
for
example,
Amazon
s3
has
eventual
consistency
and
has,
but
it
doesn't
have,
but
if
absent
guarantees.
So
when
you're
using
Delta
S
3,
there
are
special
configurations
that
you
have
to
set,
which
is
in
which
is
all
documented
in
the
Delta
Lake
docks.
So
please
read
those
when
you're
working
on
local
file
system
or
Hadoop.
A
B
Rock
on
so
for
folks
who
are
at
who
would
like
to
ask
questions?
Could
you
do
your
best
to
please
put
those
questions
in
the
Q&A
session
as
opposed
to
in
the
chat,
so
that
way
these
are
for
us
to
track
them
all.
Ok,
so
let
me
start
taking
a
look
at
it.
Actually,
hey
Broc,
there's
a
really
good
question:
I'm,
let's
start
with
this
one,
do
you
recommend
Delta,
like
400
LTP,
like
workloads.
A
Great
question
I
would
definitely
not
recommend
Delta
Lake
for
OLTP.
You
can
think
of
Delta
more
for
OLAP
use
cases
and
like
Big,
Data,
so
think
about
it.
This
way,
you
don't
want
to
insert
one
record
at
a
time
or
update
one
record
at
a
time.
You
want
to
batch
things
as
much
as
possible
so
to
provide
this
bashing,
you
can
use
some
centralized
message
buffer,
like
Kafka
or
Kinesis
or
event
hubs
to
actually
batch
up.
You
know
like
10,000
changes
or
a
hundred
thousand
changes.
A
C
B
A
B
A
B
B
But
do
note
the
fact
that
recall
that
the
the
transaction
long
basically
hasn't
changed
okay,
but
the
underlying
data
has
so,
in
other
words,
when
you
run
a
query
against
a
version
that
you
see
in
the
transaction
log,
but
no
longer
the
file
no
longer
exists
in
the
system.
What'll
end
up
happening
is
you'll
get
an
error.
Okay.
Now,
in
this
case,
this
is
a
desired
error,
because
you're
actually
doing
exactly
what
happened.
B
B
B
I
would
not
try
to
run
this,
like
you
know,
for
like
three
years
worth
of
data
right
if
you're
gonna
do
this
for
three
years
of
data,
I
would
actually
snapshot
those
tables
out,
save
them
to
cold
storage
or
something
like
into
that
and
then
like,
for
example,
a
good
example
will
be
like
financials
right.
You
know
like
financial
reporting,
okay,
fine,
fair
enough,
then
I
take
a
snapshot
of
the
table
on
the
31st
or
the
first,
depending
on
how
you
do
it
or
all
the
fifth.
B
So,
let's
see,
there's
we've
probably
got
time
for
about
two
more
questions:
okay,
Brock,
because
I
don't
want
to
close
things
up.
Let's
see,
okay,
a
good
question
that
we'll
dive
into
deeper
next
session,
but
I
did
want
to
go
ahead
and
call
it
out
now.
If
there
is
a
schema
change
apart
from
adding
column,
is
there
any
other
way
than
to
reprocess
older
data?
With
the
new
schema,
that's.
A
A
You
know
allow
certain
data
type
changes
which
parky
allows
and
that's
only
for
you
know
short
Speights
and
int
are
stored
as
it's
within
park',
so
you
can
up
cast
from
one
to
the
other,
but
you
cannot
change
from
in
to
long.
For
example,
in
those
cases
party
encodes
these
values
differently.
Therefore
you
would
have
to
either
overwrite
your
entire
table,
or
you
can
also
decide
to.
You
know,
create
a
view
on
top
of
your
table.
That
would,
you
know,
emit
certain
columns
or
whatnot
yeah.
B
Makes
a
ton
of
sense,
okay,
so
then
the
final
question
when
using
Delta
Lake
with
structured
streaming,
we
sort
of
alluded
to
this
before.
But
let's,
let's
call
out
specifically,
you
can
end
up
with
a
lot
of
small
files,
just
as
we've
seen
so
what
what
do
you
think
is
the
best
way
to
handle
all
of
this
then
yep.
A
As
Danny
showed
in
the
demo,
we
have
the
ability
to
read
from
your
Delta
table
and
right
back
to
your
Delta
table
with
the
option
of
Delta
data
change
false.
So
this
just
tells
Delta
that
you
know
this
transaction
is
happening.
It's
only
you
know
rearranging
the
data
in
a
you
know
different
layout,
it's
not
changing
any
of
your
data,
so
this
has
like
a
lot,
looser
isolation,
guarantees
or
when
you
know
committing
this
transaction.
So
you
can
use
this
data
change
false
to
reorganize
your
data,
as
well
as
within
data
bricks.
We
have
more.
A
You
know
we
try
to
make
this
process
simpler
with
you
know,
optimize
as
a
function
which
you
know
can
do
other
things
underneath
the
hood
but
yeah
this
like
data
change.
Falls
the
asset
transactions
is
the
key
here.
You
can
really
organize
rewrite
your
data
in
any
way
that
you
want.
That
could
give
you.
You
know
better
data
skipping
guarantees,
better
indexing
guarantees
and
whatnot.
B
Perfect
well
I
think
that
covers
it.
For
today,
I
realize
there
are
a
bunch
of
other
questions
that
we
haven't
had
a
chance
to
answer
saying
this.
This
video
will
be
propped
up
on
to
the
YouTube
channel
database
YouTube
channel
and
from
there
you're
more
than
welcome
to
continue
conversing
with
us
and
asking
questions
there
we'll
go
and
do
our
best
to
answer
those
questions
there.