►
Description
Online Tech Talk hosted by Denny Lee, Developer Advocate @ Databricks with Andreas Neumann, Staff Software Engineer @ Databricks
Link to Slides - https://github.com/dennyglee/databricks/blob/c42c3939e2b6ae8a97d70207554105ffaaf26206/notebooks/Users/denny.lee%40databricks.com/Delta%20Lake/Delta%20Lake%20-%20Enforcing%20and%20Evolving%20the%20Schema.pdf
Link to Notebook - https://github.com/dennyglee/databricks/blob/master/notebooks/Users/denny.lee%40databricks.com/Delta%20Lake/Diving%20into%20Delta%20Lake%20-%20Enforcing%20and%20Evolving%20Schema.py
Link to Diving into Delta Lake Part 1: https://www.youtube.com/watch?v=F91G4RoA8is
Link to Online Meetups Playlist: https://dbricks.co/youtube-meetups
Abstract:
Data, like our experiences, is always evolving and accumulating. To keep up, our mental models of the world must adapt to new data, some of which contains new dimensions – new ways of seeing things we had no conception of before. These mental models are not unlike a table’s schema, defining how we categorize and process new information.
This brings us to schema management. As business problems and requirements evolve over time, so too does the structure of your data. With Delta Lake, as the data changes, incorporating new dimensions is easy. Users have access to simple semantics to control the schema of their tables. These tools include schema enforcement, which prevents users from accidentally polluting their tables with mistakes or garbage data, as well as schema evolution, which enables them to automatically add new columns of rich data when those columns belong. In this webinar, we’ll dive into the use of these tools.
In this webinar you will learn about:
- Understanding table schemas and schema enforcement
- How does schema enforcement work?
- How is schema enforcement useful?
- Preventing data dilution
- How does schema evolution work?
- How is schema evolution useful? Databricks is proud to announce that Gartner has named us a Leader in both the 2021 Magic Quadrant for Cloud Database Management Systems and the 2021 Magic Quadrant for Data Science and Machine Learning Platforms. Download the reports here. https://databricks.com/databricks-named-leader-by-gartner
A
Going
to
be
talking
about
enforcing
and
evolving
the
schema
with
Delta
Lake.
This
is
part
two
of
our
diving
into
Delta
Lake
series,
and
my
name
is
andreas
Nolan
and
my
co-host
is
Danny
Lee.
So
very
quick
introduction
about
myself
I'm
a
software
engineer
at
data
breaks
I
dedicate
almost
all
of
my
time
to
Delta
Lake
and
also
to
how.
C
C
C
Thank
ondrea's
hi
everybody.
My
name
is
Danny
Lee
I'm,
a
developer
advocate
at
data
bricks
have
been
working
with
patchy
spark
since
0.5
0.6
days
before
this
I
was
the
senior
director
of
data
science.
Engineering
at
concur
and
I'm.
Also
former
Microsoft
II
being
up
here
in
the
Seattle
area.
Woohoo
go
Hawks,
that's
right,
I
called
it
out
everybody,
so
as
Microsoft
II
I
was
involved
with
the
cosmos,
DB
team,
HD
insight,
isotope
and
also
with
the
sequel
server
team,
some
a
longtime
database
guy.
C
So
hence
the
reason
why
Delta
Lake
is
awesome
and
then
master's
biomedical
informatics
at
all.
Wait
to
see
you
and
I'm
not
as
interesting
as
Andreas
I
just
have
a
BS
in
physiology
at
McGill.
Basically,
Asian
parents
was
supposed
to
be
a
doctor
turned
out.
That
was
a
really
bad
idea.
So
there
you
go
I.
A
C
B
C
A
A
What
we're
gonna
talk
about
is
that
data
is
constantly
evolving
and
constantly
changing
alright.
Why
is
that?
Because
it
reflects
all
the
things
that
we
experience.
It
reflects
the
business
problems
and
the
business
requirements
that
we
have
so
as
those
change
the
structure
of
our
data
changes
right
now,
when.
A
B
A
B
A
A
Table
schema,
the
table
schema,
describes
the
structure
of
our
data,
but
in
apache
spark,
for
example,
every
data
frame
has
a
schema
and
if
you
use
Delta
Lake
as
your
storage
formed,
then
this
scheme
of
that
data
becomes
the
schema
of
the
table
and
it
is
saved
in
a
JSON
format.
Inside
the
transaction
log
we
saw
the
transaction
log
last
week
right
and
what
does
this
key
might
look
like.
So
it's?
Basically
it's
it's
a
of
fields.
Right.
Each
field
has
a
name
oops.
A
Each
field
has
a
type
and
it'll
also
say
whether
it's
novel
alright,
if
it's
not
able
that
means
the
field
doesn't
have
to
be
present,
but
it's
not
level
as
faults
than
every
record.
That's
written
to
the
table
is
required
to
have
this
this
column
or
this
field.
Okay.
So
knowing
what
the
schema
is,
let's
talk
about.
What
schema
enforcement
is
schema.
Enforcement,
also
known
as
schema
validation,
rejects
any
rights
to
a
table
that
do
not
match
the
tables
EEMA.
So
what
does
that
mean?
A
A
Isn't
present
in
the
in
the
table
schema
then
the
table
will
not
accept
this
data.
It's
okay!
On
the
other
hand,
if
the
data
that
I'm
writing
does
not
contain
all
the
columns
right,
some
of
the
columns
can
be
missing
and
then
what
happens
is
in
that
data
that
I'm
writing.
These
columns
will
be
assigned
null
values
and
that's.
A
In
addition,
you
cannot
have
columns
that
have
a
different
data
type
than
what's
defined
in
the
table
schema
right.
So,
for
example,
if
my
table
says,
oh,
this
column
has
type
string,
but
the
data
that
I'm
trying
to
write
has
type
integer.
Then
schema
enforcement
will
fail
this
right
and
it
will
throw
an
exception
and
no
rights
will
happen.
A
Another
pretty
important
and
tricky
case
here
is:
you
cannot
have
column
names
that
differ
from
the
tables
email
only
by
case.
So
what
does
that
mean?
Let's
say
you
have
a
you:
have
a
column
foo
with
a
capital
F.
Then
you
cannot
have
another
column
where
the
F
is
is
this
is
a
small
caps
letter,
and
why
is
that
there's
a
little
bit
of
background
here.
A
Preserving
so
it'll
simply
pass
through
the
case
that
you
give
it,
but
when
it
stores
the
schema
unit,
it
won't
allow
you
to
have
two
columns
that
have
the
same
name
except
for
case,
and
this
is
simply
to
prevent
like
potential
mistakes
and
to
prevent
unexpected
things
that
that
might
happen
to
your
data.
If
you're.
B
A
Aware
of
the
of
the
case,
sensitivity
issues
alright,
so
that
was
schema
enforcement
and-
and
why
is
schema
enforcement
useful?
What
what
are
some
situations
where
I
need
this
right,
and
this
is
it's
basically
anytime,
when
I
run
production
systems
that
really
depend
on
the
fixed
structure
of
the
data
that
they're
really
alright,
so,
for
example,
machine
learning
algorithms,
when
I'm
training
my
model,
it
expects
the
exact
kind
of
of
data
there
and
if.
A
So
in
order
to
do
that,
because
your
data
typically
arrives
in
your
data
center
or
in
the
cloud
with
schema
variation,
a
lot
of
people
build
pipelines
that
sort
of
employ
a
multi
hop
approach.
So
the
first
hop
will
just
like
ingest
raw
data.
The
next
one
will
filter
out
bad
data
and
the
next
one
might
canonicalize
the
schema
so
that,
like
in
the
end
where
you
have
your
gold
tables
at
all,
the
records
in
those
tables
conform
to
the
expected
schema.
A
Okay.
So
now
that
we've
talked
about
schema
enforcement,
let's
talk
about
evolution
right,
so
schema
enforcement
was
a
was
a
way
that
allows
us
to
fix
the
schema
of
our
data.
Schema
evolution
allows
us
to
change
the
schema
of
our
data
in
a
very
controlled
way,
all
right,
so
it
allows
you
to
change
the
table
schema
to
accommodate
for
data
that
has
been
changing
over
time
right
and
most
commonly.
A
A
B
A
A
Merge
schema
as
true:
we
can
basically
make
any
schema
changes
that
are
reads
compatible
and
what
does
read
compatible
mean.
It
means
that
existing
data
in
the
table
can
still
still
be
read
according
to
the
new
schema
all
right,
and
so
we
do
this
when
we
append
or
overwrite
the
table
and
the
following
types
of
changes
are
are
available
here.
So
we
can
add
new
columns
all
right
and
the
old
data
in
the
old
data.
That's
already
in
the
table.
Those
columns
will
simply
be
null.
A
We
can
change
the
type
of
of
a
column
from
non
malleable
to
inaudible
all
right.
So
that's,
basically
that's
a
relaxation
of
the
of
the
existing
schema
right,
and
so
the
old
data
will
still
fit
and
we
can
do
up
casts
right
where
we
go
from
smaller
type
to
a
bigger
time,
so
any
bites
can
be
represented
as
a
short
or
short
can
be
represented
as
an
integer.
So
the
old
data
can
still
be
read
there.
A
Stronger
form
of
schema
evolution
and
for
this
form
of
evolution
you
would
use
the
option
override
schema
and
that
allows
us
to
do
schema
changes
that
are
not
read
compatible
with
existing
data.
So
typically
we
do
this
when
we
override
the
data
right,
because
if
we
only
append,
then
the
old
data
in
the
table
becomes
useless.
But
if
we
override
all
data,
then
we
can
change
the
schema.
So
what
types.
A
Changes
can
we
make
here,
so
we
could
drop
a
column,
for
example,
or
we
could
change
the
data
type
of
a
column
right
so
something
that
was
previously.
A
string
can
now
be
an
integer
and
that
wasn't
possible
with
merge
schema
and
we
can
also
rename
columns,
and
even
if
we
just
change
the
case
of
columns,
this
is
all
now
allowed
if
we
do
override
schema.
A
All
these
demos-
oh
excuse
me.
So
all
this
will
be
available
for
you.
You
can
try
and
run
these
notebooks
in
your
own
clusters
and
data
breaks,
and
if
you
don't
have
a
data
breaks
account,
you
can
use
data
breaks,
Community,
Edition,
right
and
all
of
this
will
be
available
after
this
webinar.
The
data.
B
A
Okay,
here's
a
very
quick
intro
of
Delta
Lake
and
I
am
NOT
gonna
go
through
all
of
this.
The
two
things
that
are
important
to
us
here
in
this
talk
is
that
it's
asset
transactions,
so
we
know
that
rights
either
go
through
or
they
or
they
fail,
and
so
our
data
will
always
have
integrity,
and
the
really
important
part
here
is
the
schema
enforcement
and
the
schema
schema
evolution.
We
will
also.
B
A
To
begin
with,
we
want
to
show
how
this
was
would
look
like
if
you
did
not
have
dealt
a
leg,
so
Delta
Lake
uses
park'
files
as
its
default
storage
font,
and
so,
let's
just
play
with
some
data
just
with
park',
and
the
first
thing
we're
gonna
do
here-
is
we're
gonna
download
this
data
okay.
So
the
data
is
now
here
it's
in
this
particular
location
here
and
it
was
downloaded
to
this
location
and
that's
the
data
that
we're
going
to
be
working
with
now.
A
So
the
first
thing
I'm
doing
here
is
just
a
little
bit
of
setup,
importing
a
couple
of
things
and
setting
some
options
and
creating
a
working
directory
for
this
experiment.
Okay,
and
so
that
has
happened.
What
I'm
doing
now
is
I'm
gonna,
create
a
park,
a
table
right
and
the
way
I'm
gonna
create.
This
is
I'm.
Gonna
read
this
data
that
we
just
downloaded
and
I'm
not
gonna,
do
much
with
it.
I'm
just
gonna
write.
A
So
this
is
all
running
okay,
so
this
went
pretty
quickly
and
if
we
look
at
this
table
now,
let's
just
look
at
the
first
20
records
here.
We
can
see
that
this
table
has
a
schema
right,
that
every
row
here
has
a
loan
ID,
it
has
a
funded
amount,
they
all
appear
to
be
a
thousand
and
then
it
has
the
paid
amount.
This
is
how
much
has
been
paid
off
and
then
it
also
has
the
states
where
zooom
was
funded,
and
this
looks
pretty
uniform
and
and
pretty
good
data.
Let's
see.
A
B
A
That
we're
gonna
run
a
streaming
query:
we're
going
to
do
structured
streaming
and
this
stream
that
we're
going
to
run
is
going
to
generate
random
data.
That
also
has
long
IDs
and
alright.
So
for
this
I'm
just
defining
a
couple
of
utility
functions
here:
here's
something
that
generates
the
states
randomly
the
main
method
here.
Is
this
one
right,
so
what
it
does
is
it
uses
the
spark
format
rate?
What
so,
what
rate
does?
A
Is
it
simply
generates
rows
every
second,
it
generates
I
think
here
it
generates
five
rows
and
each
row
will
have
a
timestamp
in
the
value
and
then
you
can
use
those
things
to
create
new
columns
from
that
right.
So
here
we
we
take
the
that
value
and
add
10,000
to
it,
and
then
we
have
a
new
loan,
ID
and
so
on,
so
we're
basically
generating
these
four
columns
here
that
were
in
the
schema
of
the
table
that
we
saw
up
here,
okay,
so
yeah.
A
B
A
Run
this
query
all
right,
so
this
should
now
every
10
seconds
it
should
add
some
data.
Let's
look
at
this
query.
It's
running
it's
still
initializing,
it
seems,
but
pretty
soon
we
should
be
seeing
some
change
here.
Okay,
so
now
it's
processing
data.
We
can
see
that
here
and
so,
let's
look
at
this
table.
Let's
count.
B
A
See
that
the
number
of
records
in
this
table
is
steadily
going
up,
so
it
looks
like
this
streaming.
Query
is
working
and
it
does
what
it's
supposed
to
do,
but
wait.
We
had
fourteen
thousand
seven
hundred
and
five
rows
in
this
table
and
where
did
those
go?
I
suddenly
only
have
a
hundred
and
forty
five.
This.
A
Well,
let's
look
at
this
data
again,
let's
see
so,
let's
just
look
at
a
few
of
the
records.
Okay,
oh
my
yeah,
so
I
see
that
somehow
my
data
is
different.
This
data
has
different
schema.
It
has
two
extra
columns,
it
has
a
timestamp
and
it
has
a
value
okay.
So
where?
Where
do
these
come
from?
It's
it's.
This
is
confusing,
but
let's
see
well,
let's
look
at
let's
look
at
this
code
again
and
if
we
look
at
this
closely,
we
see
that
it
uses
this
format.
A
A
So
this
didn't
work
very
well
all
right
and
now
I
want
to
show
you
how
Delta
Lake
can
protect
us
from
this
kind
of
situation,
where
we
are
unknowingly,
dropping
data
okay,
so
we're
gonna
do
a
very
similar
setup
as
we
did
for
Pocky,
but
this
time
we're
just
going
to
do
it
with
Delta
right.
So
we're
reading.
B
A
A
Looked
like
when
we,
when
we
did
Park
it
so
just
as
a
quick,
deep
dive
into
how
Delta
Lake
stores
the
schema.
Let's
look
at
the
files
that
are
actually
in
the
directory
that
represents
this
table.
So
here
we
have
a
parkade
file
and
this
is
the
one
that
was
written,
but
we
also
have
an
extra
directory.
That's
the
Delta
log
and
the.
A
Contains
deltas
transaction
log,
so
let's
look
at
the
files
that
are
here
so
this
should
contain
commits
because
we've
only
run
one
little
batch
job
against
this
there's.
Only
a
single
coumadin
that
commit
is
represented
by
this
JSON
file.
All
right!
So
that's
look
at
this
Jason
file.
Okay,
so
this.
B
A
This
is
pretty
complex,
Jason.
Let's,
let's
look
at
this
in
a
different
way,
because
spark
has
a
way
to
read
this
s:
Jason.
Okay,
now
we
can
see
the
structure
of
this
and
we
can
see
that
this
is
a
pretty
nested
and
pretty
original
structure.
It
has
an
ads
field
which
says
what
information
was
added
in
this
commit
it
has
a
general
commit
info
and
it
also
has
two
meta
data
in
a
protocol.
So
let's
look
look
at
these
things
in
detail.
A
A
Is
indeed
the
idea
of
my
notebook
and
we
can
see
as
timestamp
and
we
can
see
my
user
ID
and
my
email
address,
so
that's
that's
pretty
cool.
That
gives
us
a
lot
of
information.
Now,
let's
look
at
so
what
information
was
added?
What
data
was
added
in
this
commit
and
what
we
see
here
is
there?
Was
they
don't
change?
There
is
a
modification
timestamp.
We
also
have
a
path
and
this
path
matches.
B
A
We
saw
in
the
directory
listing
up
here
all
right.
That's
exactly
this
path
that
belongs
to
this
commit
and
we
see
some
stats
all
right.
So
the
statistics
help
the
statistics,
help
Delta
optimize
queries
and
we
see
that
these
statistics
it
exists
for
all
the
four
columns,
all
right:
the
loan
ID
the
funded
amount
of
paid
a
month
in
the
in
the
state.
Now,
let's
look
at
the
metadata
and
a.
B
A
A
creation
time
and
it
has
the
former's
of
the
file
but
most
importantly,
it
has
the
schema
stream.
So
this
is
a
parquet
or
an
avro
scheme,
a
string
and
it
describes
exactly
the
fields
that
are
in
this
table.
Alright,
so
there's
a
loan
ID
as
a
funded
amount,
there's
a
paid
amount
and
there's
the
state.
Ok.
So
now.
A
Records
the
schema
of
the
table
in
its
metadata:
let's
run
a
streaming
count
on
this
table,
because
what
we're
gonna
do
next
is
we're
gonna
start
appending
more
data
to
this
table
and
so
running.
This
streaming
count
will
continuously
count
the
table
and
update
these
counts
in
real
time,
and
the
first
thing
we're
going
to
do
is
we're
just
gonna
write
data
in
the
same
way
that
we
wrote
it
to
Parque.
A
This
is
the
same
function
that
we
call
previously.
It's
just
using
table
format
Delta
now,
ok,
and
what
we
see
here
is
that
we
get
an
analysis
exception
right,
so
spark
refuses
to
run
this
query
and
it's,
and
why
does
it
do
that?
Because
the
table
schema
doesn't
match
right
and
it
actually
tells
us
the
difference
between
the
two
schemas
right?
That's
these
is
these
two
columns
well.
B
B
A
Now
we
know
that
we
have
to
fix
this
query
because
it's
it's
incompatible.
So
one
thing
we
can
do
with
this:
we
can
just
simply
add
a
projection
as
the
last
statement
in
the
query,
which
selects
only
the
four
columns
that
were
interested
in
yeah.
So
let's
define
this
and
run
it
again
with
the
modified
query.
So
now,
as
this
query
is
running
remember,
we
started
this
query
up
there
and
that
is
constantly
counting
how
many
records
are
in
the
table.
So
let's
look
at
the
status
of
this
thing
and
let's
see
so.
A
B
A
Yeah,
we
also
see
this
as
a
larger
count
right.
The
nice
thing
about
Delta
is:
you
can
have
many
concurrent
jobs
running
against
the
same
table.
They
will
not
conflict
with
each
other,
they
don't
get
in
each
other's
way
and
they
can
be
streaming
and
batch
and
you
can
mix
them
as
you
like.
Alright,
so
schema
enforcement
worked
and
now
we're
gonna.
Stop
the
streams
for
this
and
we're
going
to
try
to
do
some
experimentation
with
schema
evolution.
A
Okay,
so
before
I,
do
this
I
just
want
to
remember
how
many
commits
I
had
in
this
table,
so
the
last
commit
in
this
table
after
I
ran
that
last
three,
it
was
number
seven
okay
and
we
can
see
that
if
we
list
we
see
that
in
the
Delta
log
they're
actually
exactly
seven
commits
okay.
So
we're
going
to
remember
this
number
seven
and
now
again
we're
gonna
run
the
swinging
count
to
see
the
counts,
updating
in
real
time
now.
A
A
At
the
actual
error-
and
so
the
error
tells
us-
the
schemas
are
different,
but
it
also
gives
us
a
hint
here.
It
says:
well,
if
you
want
to
have
schema
migration,
which
is
another
term
for
schema,
evolution,
use
this
option.
So,
let's,
let's
try
that
let's
do
that
and
modify
our
query
to
use
this
option
right
and
so
here
we're
just
modifying
this
function.
To
add
one
option:
merged
EEMA
equals
true
and
then
am
I
gonna.
Try
this
again.
B
A
This
query
is
initializing,
it's
now
running
and
let's
go
up
here
to
the
streaming
Comden,
let's
see
what's
happening
here
so
accounting.
Oh,
this
query
failed,
and
why
did
this
fail?
Now?
Oh,
it
detected
a
schema
change,
so
it
it
detected
that
there's
two
additional
columns
in
the
table.
This
is
interesting
right.
So
when
the
schema
changes,
we
have
to
restart
our
our
streaming
query
because
we
can't
just
dynamically
change
the
schema.
While
it's
running
and
the
error
message
actually
tells
us
hey,
why
don't
you
try
restarting
the
query?
A
This
will
refresh
the
schema
and
then
it
should
work
again.
So
let's
try
running
this
this
counter
again,
and
so,
when
we
start
this
as
a
new
query,
then
it
should
work.
So
let's
look
at
this
and
yeah
so
now
it
can
count
and
we
should
see
the
consequence
in
the
meantime,
I'm
also
gonna.
Do
a
batch
query
just
to
see
that
yeah
that
matches
right
the
counts
here
go
up
and
the
counts
here
have
also
been
higher.
So
we
saw
that
schema.
A
Evolution
worked
right
at
the
moment
that
I
added
merged
schema
as
an
option.
I
was
actually
able
to
dynamically
change
the
schema
as
the
shape
of
my
data
changed
alright.
So
after
we've
seen
this
I'm
gonna
stop
my
streams
in
and
by
the
way.
If
you
run
this
on
Community
Edition
make
sure
you
always
stop
the
streams
when
it's
in
the
notebook,
because
otherwise
you
might
quickly
run
out
of
quota
in
your
in
your
clusters,
all
right.
A
So
now,
after
we've
done
this
schema
evolution,
let's
take
a
look
at
the
history
of
this
table
and
how
it
changed
over
time,
as
the
schema
changed
so
well.
Look
at
all
the
files.
Again
we
see
that
there's
a
bunch
of
of
new
commits
now.
So
here
number
seven.
That
was
the
last
commits
that
I
had
before
I
started,
schema
evolution.
Well
then,
we
see
there's
a
two
minute
gap
here,
so
commit
number
eight
I
would
think
must
be
the
first
commit
that
has
the
schema
change
right
and
then
from
nine
on.
A
All
these
commits
must
have
the
new
schema.
So
we
can
also
see
this
in
the
Delta
table.
History
Delta
actually
has
an
API
that
lets.
You
see
the
history,
and
here
we
can
nicely
display
that
we
can
see
all
the
commits
and
we
can
see
that
timestamp.
We
can
see
things
like
the
query.
Id
the
note
book
ID
and
an
interesting
thing
that
we
can
see
here
is
between
commit
number
seven
and
commit
marade.
The
query
ID
is
changing,
and
why
is
that?
Because
here
I
started
writing
with
a
new
query.
A
A
Number
before
my
schema
change,
I
think
it
was
seven
right
and
so
I'm,
just
adding
one
for
this
to
get
the
schema
change
itself
and
then
everything
after
that
should,
which
is
nine
and
greater,
should
have
the
news,
email
and
now
that
I
have
this
I'm.
Just
gonna
read
this
three:
these
three
commits
into
the
JSON
format
and
then
I'm.
We
can
look
at
these
commits
in
detail.
So
first,
let's
look
at
the
commit
before
the
schema
change.
A
What
we
see
here
is
it
has
a
has
a
bunch
of
data
changes
right
and
if
we
look
at
an
individual
one,
we
can
see
it
has
here
32
records
in
this
file,
and
it
has
statistics
for
the
four
fields
that
were
in
the
table
before
the
schema
change
all
right.
So
that's
exactly
what
we
expected,
and
now
we
look
at
the
commit
that
maybe
the
kheema
change.
B
A
So
this
also
has
a
data
change
but
wait.
This
is
a
data
change,
but
it
has
zero
records
like
why.
Why
are
there
no
records
in
this
commit
if
it
says,
there's
a
data
change?
Well,
the
the
beef
of
this
commit
is
actually
in
its
metadata
rather
than
to
see
what
really
happened
in
this
form.
It
is,
we
have
to
look
at
the
metadata
and
if
we
look
at
the
metadata,
we
will
see
that
this
commit
has
a
schema,
and
in
this
schema
we
have
the
two
new
fields.
A
We
have
the
timestamp
and
we
have
the
value
right.
So
what
we
see
here
is
that
when
we
do
a
schema
evolution,
Delta
Lake
adds
it
basically
adds
an
empty
commit
point
that
all
it
does
change
the
schema,
but
it
doesn't
really
add
data
alright,
and
now
we
can
look
at
the
next
limit.
We
can
look
at
the
data
that
it's
added
and
this
now
again
has
records
in
the
files
that
it
wrote,
and
it
also
has
statistics
for
the
new
fields.
So
we
can.
A
A
Now
that
was
the
first
time
we
wrote
to
it
and
then
the
second
one
will
be
the
version
of
the
table
just
before
the
schema
change
and
the
third
one
will
be
now
the
latest
version
of
that
table,
and
if
we
count
all
the
records
at
all
these
three
different
times,
we
will
see
that
yeah.
Initially
it
was
fourteen
thousand
seven
hundred
and
five
at
the
time
of
schema
change.
We
had
more
and
now
the
current
version
has
even
more
records
in
it,
and
that
was
exactly
what
we
expected
the
nice
thing
is.
A
We
can
actually
also
look
at
the
data
and
if
we
look
at
the
data,
even
though
the
schema
of
the
table
has
changed
right,
so
our
schema
now
includes
two
additional
fields.
These
two
fields
were
not
present
in
this
commit
right
in
C
underscore
before
that
was
number
seven.
So
if
we
read
this
data
on
now,
we
will
still
see
the
old
data
and
we
will
see
it
with
the
old
schema
right.
A
Now
we
go
to
the
time
just
before
the
schema
changed,
and
if
we
look
at
this
data,
it's
still
no.
This
is
the
time
when
the
schema
changed
right.
This
is
exactly
commit
number
eight,
and
what
we
see
now
is
that
we
see
the
old
data
right.
This
is
exactly
the
same
data
that
we
saw
before,
but
now
we
see
two
additional
columns,
all
the
values
here
are
now,
and
why
is
that?
A
Because
we
haven't
written
any
data
that
actually
has
these
columns
right,
and
so
they
default
to
the
null
value,
and
that's
that
this
is
exactly
what
we
expected
now.
If
we
go
to
the
latest
version
of
this
table,
or
even
if
we
just
go
to
the
first
version
of
the
table
right
after
the
schema
change
right,
this
was
commit
after
then,
we
will
actually
find
records
where
the
timestamp
is
not
now,
and
we
was.
We
will
see
that
here
you
see
here,
and
so
now
we
have
actual
data
with
the
new
schema.
A
A
A
A
A
We
have
data
ingestion
companies
and
we
also
have
the
big
cloud
vendors
right,
Google
data
proc
supports
this,
as
your
synapse
II
supports
this,
and
so
pretty
much.
No
matter
where
you
are,
you
will
find
partners
who
will
support
you
in
using
Delta
and
if
that
hasn't
convinced
you
there
is
a
bunch
of
companies
that
are
already
using
Delta
and
it's
it's
a
lot
more
than
the
ones
shown
here.
This
is
just
a
little
sample.
A
So
if
you
want
to
join
the
club
and
also
start
using
they'll,
something
ask
yourself
this
question:
how
do
I
use
it?
Well,
it's
it's
very
easy.
So
if
you
have
existing
spark
jobs
or
spark
notebooks
or
pipe
pipe,
I
spark
notebooks,
all
you
need
to
do
is
you
need
to
add
Delta
to
your
packages
or
to
your
dependencies.
So
I
think
you
do
that
on
the
command
line
for
PI
spark
or
the
spark
shell
or
you
do
that
in
Maidan.
A
A
A
This
is
part
of
a
whole
series
of
talks
and
I
encourage
you
to
subscribe
and
visit
us
again
next
time.
The
URL
is
down
here
and
if
you
want
to
become
part
of
the
community
and
build
your
own
Delta
Lake
go
to
http
Delta
dot
IO.
This
is
where
the
open-source
community
lives
and
with
that
I
want
to
open
it
up
to
questions.
C
Perfect
Thanks
address
this.
That
was
an
excellent
present
day,
so
we
have
about
10
minutes
or
so
to
answer
questions
so
I'm
gonna
start
with
answering
some
of
the
Q&A
here
to
give
Andrew
a
little
time
to
breathe.
Andreas
meanwhile
also
refer
to
the
Q&A
questions
or,
if
you'd
like
me
to
I,
can
ask
various
questions
to
you
that
you
know
for
whichever
one
you
like
to
do,
but
I
will
start
with
a
quick
question,
which
is
one
person
asks.
C
The
question
is
schema
evolution
available
in
open
source,
Delta
Lake,
and
is
there
any
difference
between
the
Delta
Lake
in
data
bricks
versus
the
open
source,
Delta
Lake?
So
as
a
quick
call
out,
while
we
did
this
presentation
in
data
bricks,
this
actually
everything
we're
showing
is
actually
open
source
available.
There
is
actually
no
difference
between
or
I
shouldn't,
say
no
difference,
but
the
goal
is
as
as
will
be
part
of
Delta
like
one
dot.
C
Oh,
there
will
be
no
difference
between
the
open
source
and
mat
or
managed
Delta
like
the
one,
that's
with
data
bricks,
because
all
of
the
api's
and
those
features
are
actually
going
to
be
exactly
the
same
okay.
So
it
doesn't
that's
the
absolutely
the
goal.
The
quick
call
out
is
that
there
are
some
potential
differences
for
data
bricks
itself
in
terms
of
the
management
of
it,
but
in
terms
of
the
actual
functionality
and
the
API
is
no
there's
gonna,
be
there
actually
is
no
difference
and
related
to
this
question.
C
There
was
a
question
concerning:
why
didn't
we
do
this
using
the
sequel,
syntax?
That's
part
of
the
reason
why?
Because
once
spark
3.0
it
becomes
available,
we
can
show
you
running
all
of
this
in
using
the
sequel
syntax
instead
of
using
the
API
syntax.
It's
just
that
we
were
we're
dependent
on
some
of
the
features
that
are
specific
to
spark
3.0
in
order
to
be
able
to
support
that.
Okay,
so
just
want
to
call
that
out
so
andreas.
C
Right,
that's
right!
Okay,
no
problem!
So
then
you
know
what
I
will
ask
you
some
questions,
and
you
can
just
tell
me
if
you'd
like
me
to
answer
them
or
if
you
want
to
go
ahead
and
answer
them,
how's
that,
sir
all
right,
so
the
the
the
the
quick
call-out
is
that
if
I
understood
this
correctly
is
our
enforcement
and
evolution
in
this
case
mutually
exclusive.
A
B
C
A
Yeah
I
mean
the
lineage
is
implicit
in
each
of
the
Delta
Lake
commits
in
the
transaction
log
right.
When
we,
when
we
looked
at
this
JSON
structure,
we
could
see
that
every
commit
has
lineage
in
it.
It
can
tell
you
what
was
the
notebook
idea.
What
was
the
spark
job
idea?
What
was
the
user
ID?
What
was
the
email.
B
C
And
just
to
add
to
your
point
right
like
for
don't
forget
that
when
you're
working
with
Park
a
the
park
itself
contains
the
schema
per
se,
but
the
problem,
this
is
the
reason
why
we
have
schema
evolution
and
schema
forcement,
which
is
to
say
that
okay
well
well,
Park
a
can
do
it.
The
reality
is
that
things
can
change
over
time.
So
you
need
something
that
actually
has
a
transaction
log
that
contains
all
of
the
potential
changes
that
were
that
way.
We
have
Ana
forcement
capability.
That
way
we
can
evolve.
C
We
can
say
exactly
to
the
point
of
lineage.
We
can
see
within
the
JSON
itself
in
the
metadata.
Okay
version:
zero.
In
the
case
this
particular
case.
This
is
where
we
went
ahead
and
had
the
initial
version
of
four
columns
by
version
was
a
seven
or
eight
that
one
is
the
where
the
switch
meet
and
then
now
it
has
six
columns.
So
you
can
actually
see
it
within
the
transaction
log
itself.
Basically,
okay,
a
fundament
for
you
Andreas!
Is
it
possible
to
relate
to
time
travel?
A
C
Cool,
let's
see
here's
another
one,
because
we
actually
have
a
lot
of
them,
which
is
basically
supporting
talking
about
schema,
supporting
schema
stuff.
So
I
think
you
answered
most
of
the
questions
so
I'm
just
trying
to
filter
them
out.
Oh
right,
I
think
we
already
answered
this
one
begin.
I.
Think
it's
a
good
call
out
in
schema.
Validation
is
a
mismatch
of
the
nullable
field,
allowed.
A
So
a
mismatch
of
a
nullable
field
is
not
allowed.
If
you
try
to
write
data
that
doesn't
contain
that
field,
then
Delta
Lake
will
throw
an
exception,
because
it
then
forces
that
not
know
that
non
malleable
fields
are
always
there.
If
the
field
is
not
able,
then
that's
relaxing
this
restriction.
Alright,
and
that
means
you
can
write
data
that
doesn't
have
these
columns
cool.
C
Then
related
to
that
as
well,
is
that,
in
terms
of
because
we've
been
talking
about
schema,
this
entire
time
does
Delta
like
support
schema-less
mode,
or
is
it
always
running
with
the
schema
and
I?
Think
you
sort
of
answered
that
already
with
the
first
commit
but
figured
I
would
call
this
out.
Yes,.
C
Perfect,
okay
and
then
all
right
now
related
to
errors
actually
with
Delta
Lake.
If
example,
you're
reading
a
message,
queue
and
then
or
and
you're
writing
to
it
and
then
Delta
has
an
error,
doesn't
automatically
replay
it
or
do
you
actually
have
to
handle
the
the
missing
message
now?
No,
this
is
more
of
a
unpacking.
The
transaction
log
question
versus
a
schema
question,
but
I
figured
it's
still
a
good
call-out
and
also
it
allows
us
to
shamelessly
call
out
that
do
go
to
the
playlist
and
and
review
the
the
previous
week's
unpacking.
A
C
Exactly
so,
just
to
add
exactly
to
Andrea's
point
of
view
point
here:
it
we
Delta
I
lake,
has
Optimax
optimistic
concurrency
model
because
there's
a
concurrency
control
model.
Excuse
me.
So
if
it's
able
to
allow
the
right
to
happen,
it
will
let
it
happen
right.
So,
in
other
words,
for
example,
if
there's
an
error
in
which
two
threads
are
trying
to
write
at
the
exact
same
time,
there
are.
C
There
are
our
checks
to
say
that
okay
well
you're
trying
to
write
at
the
same
time
the
same
table,
but
actually
it's
okay,
because
they're
going
to
two
different
partitions.
Okay,
then
there's
actually
errors
that
it
will
automatically
retry
for
you
right
away
and
it'll
take
care
of
it
that
that's
the
optimistic
concurrency
control,
but
exactly
to
Andreas
point.
If
there's
an
error
in
which
basically,
no
it
really
is
an
error.
C
There's
no
way
for
us
to
go
ahead
and
process
it
automatically
it'll
go
ahead
and
actually
error
out
and,
most
importantly,
it
will
not
leave
any
partial
written
files
so
exactly
to
address
point
about
at
atomicity,
it's
either
written
or
it
is
it,
so
you
can
trust
your
data.
This
goes
back
to
our
theme
of
reliable
data
likes
okay,
cool,
let's
see,
live
I.
Think
we've
got
time
for
probably
one
more
question,
so
actually
a
good
one
that
I
sort
of
like
talking
about
this.
C
C
So
the
fact
that
you
number
one
you
actually
have
a
transaction
log,
so
there
is
a
little
bit
of
overhead
for
doing
that
number
two,
because
we're
actually
going
in
writing
to
to
disk
the
the
issue
that
you
have,
of
course,
is
that,
as
you
can
see,
with
the
transaction
log
or
the
fact
that
we
can
have
different
versions,
every
single
version
of
the
data
with
the
its
different
schemas
actually
written.
So
in
other
words,
it's
not
like
I,
believe
address.
C
C
So,
in
this
particular
scenario,
probably
wasn't
that
much
of
a
performance
hit,
because
because
of
the
fact
that
we're
just
simply
adding
two
new
columns-
and
that
means
all
the
previous
data
was
just
nullable
so
yay-
it's
not
really,
not
that
not
much
of
an
impact,
but
how
about,
if
you
were
to
instead
to
do
updates
to
that
data
or
deletes
right
there.
That
means
what
is
implied
very
strongly.
C
Is
that
there's
that
many
more
versions
of
the
data
and
those
versions
are
much
larger,
so
the
the
potential
performance
impact
is
that
you
have
that
many
more
files.
Now
you
can
certainly
run
compaction
to
reduce
the
number
of
files,
but
the
reality
is
you
still
have
more
data,
okay
and
so
the
that's.
Ultimately
what
the
performance
impact
is.
So
it's
less
of
a
specific
to
scheme
enforcement
and
evolution
per
se
outside
the
transaction
overhead
and
more.
A
Data
right
and
you
just
keep
it
in
the
table
and
when
you
read
that
data
every
time
you
can
easily
morph
it
into
the
new
schema,
so
that
just
happens
on
the
fly
and
there's
almost
no
performance
penalty.
Whereas
if
you
do
an
override
schema,
then
you
you're
basically
forced
to
rewrite
to
make
a
new
copy
of
all
your
existing
data,
because.
C
Perfect,
okay!
Well,
you
know
what
I
want
to
be
cognizant
of
time,
so
I
apologize
for
any
questions
that
we
could
not
answer,
but
I
hope
you
definitely
enjoy
today's
session
and
we'll
head
right.
We
want
to
thank
andreas
butt-let
ahead,
read
head
back
to
rake
to
carrot,
so
let
her
in
the
show
everybody.
B
Thanks
so
much
Andreas
great
presentation,
thanks
any
is
well
for
joining
us
and
thanks
everyone
out
there
for
joining
us.
Just
a
quick
call
out
to
I
posted
the
YouTube
links
to
subscribe
and
then
also
to
join
our
online
meetup
group.
So
I
think
those
are
the
best
places
for
you
to
get
notified
on
all
of
our
upcoming
tech
talks
and
and
all
the
good
stuff.
We
have
plans
coming
up
in
the
next
few
months.
So
thanks
everyone
again
for
joining
and
have
a
great
rest
of
your
day.