►
From YouTube: Beyond Lambda: Introducing Delta Architecture
Description
Online Tech Talk with Denny Lee, Developer Advocate @ Databricks
Lambda architecture is a popular technique where records are processed by a batch system and streaming system in parallel. The results are then combined during query time to provide a complete answer. Strict latency requirements to process old and recently generated events made this architecture popular. The key downside to this architecture is the development and operational overhead of managing two different systems. There have been attempts to unify batch and streaming into a single system in the past. Organizations have not been that successful though in those attempts. But, with the advent of Delta Lake, we are seeing a lot of our customers adopting a simple continuous data flow model to process data as it arrives. We call this architecture, The Delta Architecture. In this session, we cover the major bottlenecks for adopting a continuous data flow model and how the Delta architecture solves those problems.
See full Getting Started with Delta Lake tutorial series here:
https://databricks.com/getting-started-with-delta-lake-tutorial-series/
Other links:
Notebook: https://dbricks.co/dlw-01
Slides: https://dbricks.co/BeyondLambdaSlides
https://delta.io
https://mlflow.org
Spark + AI Summit Discount Code: DennySAI020
Register for Spark + AI Summit: https://databricks.com/sparkaisummit/north-america Databricks is proud to announce that Gartner has named us a Leader in both the 2021 Magic Quadrant for Cloud Database Management Systems and the 2021 Magic Quadrant for Data Science and Machine Learning Platforms. Download the reports here. https://databricks.com/databricks-named-leader-by-gartner
A
Today,
what
are
the
complexities
to
build
a
simple
data
pipeline
and
why
do
these
complexities
even
exist?
And
then,
when
we
we
talk
about
these
complexities?
We've
also
talked
about
how
Delta
Lake,
why
Delta
like
and
how
it
works
and
then
talk
about
the
Delta
architecture.
This
is
a
pattern
that
we
often
see
with
data
its
customers,
but
then
in
terms
of
building
continuous
pipelines
with
Delta
Lake
and
how
we
want
to
apply
that
to
everybody,
so
the
open
source,
Delta
Lake
project
and
the
key
characteristics
and
benefits
of
this
Delta
architecture.
A
A
The
dream
of
that
person
is
to
go
ahead
and
be
able
to
process
data
continuously
and
incrementally,
as
new
data
arrives
in
a
cost-efficient
way,
without
actually
having
to
choose
between
back
and
streaming
right.
The
whole
premise
of
structured
streaming
and
spark
when
we
build
structured
dreaming
was
the
context
or
the
idea
that
you
didn't
want
to
actually
differentiate
between
how
you
dealt
with
batch
data
versus
how
you
dealt
with
streaming
data
right.
Basically,
this
concept
of
continuous
applications.
A
You
want
to
basically
say
you
know:
I,
don't
have
to
choose
between
bacter
streaming,
I
just
want
to
be
able
to
go
a
handle
both
at
the
same
time,
it's
just
a
matter
of
latency
right,
but
in
reality
this
is
actually
how
it
looks,
and
so
you
can
replace
Kafka
with
if
you
wanted,
with
Kinesis
or
with
event
hubs
or
cosmos.
Tv
doesn't
really
matter
that.
The
context
is
that
you
have
a
bunch
of
that's
coming
in
right
and
traditionally
you
have
a
backs
type
system.
A
The
job
becomes
we'll
the
job
will
exponentially
can
become
slower
because,
due
to
the
build
up
of
a
very
many
small
files,
okay,
that
that's
problem
number
one,
and
so
how
you
typically
solve
that
problems
by
going
ahead
and
switching
that
table
to
batch
and
then
basically
with
that
table,
we
create
another
table
that
basic
compacts,
that
information
every
hour
or
so,
and
then
that
batch
we
can
read
the
batch
off
of
that
for
our
AI
and
for
our
reporting.
Okay,
so
that
that's
more
or
less
the
context
here,
all
right.
So
all
right.
A
What
happens,
though,
if
late
data
arrives
right,
this
typically
means
that
they're
gonna
actually
have
to
go
ahead
and
run
batch
process,
so
you're
assuming
batch
processing,
but
also
delay
everything,
okay
and
that's
where
things
becomes
really
problematic
for
us.
So
in
other
words,
you've
got
the
table.
A
That's
coming
here
on
the
bottom
left,
then
in
the
middle
it
would
compacting
every
hour,
but
then
maybe
because
there's
proce
issues
of
stream,
the
this
table-
that's
in
the
middle
here
middle
bottom
here
has
gonna,
go
ahead
and
actually
have
to
be
delayed
in
order
for
us
to
process
that
data.
Okay,
all
right.
So
how
do
we
go
ahead
and
resolve
this
particular
problem?
A
Because
there's
gonna
be
business
requirements
where
I
have
to
be
able
to
look
at
the
data
coming
in
irrelevant
of
the
fact
that
that
there's
the
late
data,
because
there's
still
all
the
other
data
so
to
solve
that
problem?
Typically
a
build
web
better
sure.
So
this
is
what
the
top
the
top
stream
is.
So
I
still
have
the
bottom
stream.
That
gives
me
my
batch
data,
but
then
I
build
the
top
tree.
A
That's
you
know,
structure
stream
against
Kafka,
and
that
is
processing
data
on
a
regular
basis
and
then
basically
build
a
unified
view
that
the
validates
the
dated
between
the
streaming
data
and
the
batch
data.
Okay,
but
of
course,
increases
the
operational
burden
because
you
actually
have
to
go
ahead
and
now
maintain
both
a
batch
process
and
streaming
process.
A
Okay,
and
even
if
I
wasn't
so
concerned
about
that
operational
aspect,
right
I
actually
need
a
validate
right,
so
as
the
stream
is
being
processed
on
top
and
the
stream
is
being
processed
on,
the
Left
I'll
need
to
read
and
write
some
code
and
run
some
code.
That'll
actually
validate
the
data
that
in
that,
what
I'm
reading
on
the
top
and
what
I'm
reading
on
the
left
in
fact
are
the
same
values.
If
they're
not
the
same
values,
then
they're
sub
see
there's
a
problem
and
processing
somewhere
in
between
all
this.
A
A
Okay,
so
it's
as
you
can
tell
this
is
starting
to
get
more
and
more
complicated
right,
but
this
is
traditionally
what
we
had
to
do
when
we,
when
we
were
looking
at
trying
to
do
a
I'm
reporting,
but
we
actually
needed
to
go
ahead
and
look
at
both
the
streaming
data
and
making
sense
of
that
data
and
also
the
batch
data
at
the
same
time
and
then,
if
I
needed
up,
do,
updates
and
merge.
This
gets
really
complex
when
you're
dealing
with
the
table.
So
it's
this
is
just
a
high-level
summary
of
the
context
here.
A
I
don't
want
to
dive
too
deep
into
it
more
just
because
the
context
is
this
is
the
pain,
okay,
that
youyou
are
typically
working
with,
and
the
usual
question
that
we
want
to
ask
you
is:
can
this
be
simplified?
Alright?
So
this
is
where
we
see
if
we
can
simplify
to
this
point
where,
basically,
all
it
is,
is
you've
got
your
streaming
sort
your
streaming
and
your
back
sources
to
the
left.
You
use
Apache
spark
to
process
it
into
some
store
and
then
from
that
store,
go
ahead
and
do
your
AI
report.
Okay.
A
So,
what's
missing
in
order
to
be
able
to
pull
this
off?
Is
the
ability
to
read
consistent
data
while
data
is
being
written
and
the
context
is
that
don't
forget
when
you're
dealing
with
distributed
systems,
it's
typically
based
versus
asset
transactions
and
then
base
basic
base
stands
for
basically
available
soft
state.
A
But
the
most
important
aspect
is
the
e,
which
is
eventually
consistent,
and
what
that
really
means
is
that
when
you
write
the
data
to
your
storage
or
you
to
your
disk,
typically
underneath
the
covers,
whether
it's
using
Hadoop
or
using
a
cloud
storage,
there's
three
copies
of
the
data
or
more
for
that
matter.
They're
written
to
disk.
Well,
that's
great,
because
it'll
ensures
availability,
but
the
problem
is
that
you're
not
sure
if
all
three
are
written
at
the
same
time,
I
mean
a
sizable
chunk
of
the
time
it
is.
A
But
if
you
want
guarantees,
no,
there
are
no
guarantees
for
that
right.
That's
what
each
that's,
where
eventually
consistent,
so
it's
possible
for
client,
one
to
read:
node
one
and
Clyde
two
to
read
no
to
where
client
one
has
that
file
or
that
data
and
climb
to
does
not.
Yet
right
so
that's
mean
so
you're
inconsistent
right.
So
if
you
want
that
ability
to
be
able
to
read
streaming
data
and
be
able
to
read
batch
data
and
simplify
your
your
pipeline,
you're
gonna
have
to
have
consistently
know
that
that's
the
key
call
out
all
right.
A
You
want
the
ability
to
read
incrementally
from
a
large
table
with
good
throughput
right.
This
is
the
context
of
us
saying:
okay,
if
you
actually
have
to
worry
about
all
these
small
data
files,
how
do
I
go
ahead
and
handle
that
you
wanna
the
ability
to
roll
back
in
case
of
bad
writes
that
that
can
be
extremely
problematic.
You
want
to
be
able
to
replay
historical
data
along
new
data
that
arrives
so,
in
other
words,
take
all
the
history
data
about
the
back
state.
A
If
you
want
to
think
of
this
way
and
the
streaming
media
and
be
able
to
merge
it
together
or
join
it
together
or
Union
together
right,
you
want
to
be
able
to
go
ahead
and
look
at
the
data
irrelevant
of
latency.
You
just
want
to
be
able
to
treat
it
as
one
so
I
can
I
use
that
unified
view.
So
I
can
go
ahead
and
actually
make
sense
of
it
right
from
the
get-go
all
right
and
just
as
important
the
ability
handle
late
arriving
data
without
having
to
they
downstream
processing.
A
So
in
other
words,
yes,
the
data
comes
in
yes,
it's
late,
because
the
source
system
was
late,
but
that's
okay.
You
know
we're
not
going
to
need
to
delay
the
downstream
processing
for
it.
Okay.
So
the
answer,
basically,
is
this:
this
combination
of
taking
structured
streaming
plus
Delta
Lake
structure
streaming
gives
you
takes
here.
That
leads
the
aspect
for
you,
but
then
Delta
Lake
takes
to
the
aspect
of
asset
transactions.
A
A
A
That's
your
Delta
architecture,
okay,
and
so
with
the
most
important
aspects
of
this
in
that
case,
is
that
the
first
one
is
unified,
batch
and
streaming
with
a
continuous
data
flow
model,
so
in
other
words
whether
you're
processing
the
data
reading
the
data
in
streams
or
in
batch
writing
and
streams,
or
in
batch
you're
able
to
do
it.
Nevertheless,
okay,
that's
an
super
important
aspect
of
that.
Okay,.
A
That's
besides
the
point,
but
the
context
is
that
you
might
need
to
scale
that
or
in
the
scale
your
compute
right,
but
you
don't
actually
have
a
reliance
where
you
need
high-level
amounts
of
compute,
but
you
don't
really
need
a
lot
of
large
amounts
of
storage
and
because
of
that
you're
forced
to
increase
storage
at
the
same
time
with
traditional
systems.
Instead
with
the
ability
to
actually
say
no
I
need
more
storage
then
scale
that
I
need
more
computers
to
scale
that,
but
those
are
independent
processes.
Okay,
so
underneath
the
covers
I
had
talked
about.
A
Okay.
Well,
then,
in
order
for
us
to
do
this,
we
need
asset
transactions.
We
need
schema
enforcement,
we
need
time
travel
and
so
we've
talked
about
Delta
Lake.
So
how
does
it
work?
Okay,
so
before
I
explain
this
concept,
let
me
just
switch
real
quick
to
a
demo,
okay,
all
right,
so
so
I've
already
created
a
Delta
table.
A
A
A
That's
what
happens
when
you
run
demos
live
all
right,
so
we'll
get
will
kick
this
off
all
right,
and
so,
while
I
kick
this
off,
the
the
the
quick
call
up
that
I
wanted
to
make
was
that
there
is
a.
It
looks
very
much
like
your
standard,
Delta,
so
parquet
table,
except
that
there's
also
an
underscore
delta
underscore
log
folder,
okay,
so
I'm
gonna
go
ahead
and
talk
about
that.
A
Real
quick
was
that
when
I
click
this
back
off,
and
so
what
happens
here-
is
that
the
delta
underscore
log
here
is
actually
your
transactions,
okay,
a
log
all
right,
and
so
the
whether
you
have
partitioned
directories
in
in
my
example,
I,
don't
have
partitions,
but
you
could
throughout
if
I
part,
you
definitely
just
like
any
other
Park
a
table
yeah.
For
your
part,
your
partition
directories
inside
there's
a
file1
file2
parquet
right,
so
that
that's
actually
what
exists
but
what's
also
exists
in
order
to
make
this
a
Delta
table.
A
Is
this
underscore
delta
underscore
log
with
different
dot
JSON
files
inside
there?
Okay-
and
these
did
these
basically
tell
you
the
different
versions
of
the
table.
So,
as
you
add
more
and
more
of
these
things
more
data
into
the
system,
more
transactions
to
system
there
are
more
transactions,
ie
JSON
files,
JSON
files
that
show
up
to
represent
each
transaction.
There
is
a
question
to
the
from
the
audience
that
do
you
get
these
notebooks
as
well.
A
A
See
how
we're
doing
here
all
right,
let's
go
back
and
take
a
look
all
right.
There
go
okay,
perfect!
So
there's
only
because
I
just
created
this
table,
because
I
reran
everything
all
over
again.
Basically,
we
only
have
a
single
JSON
file.
Okay,
that's
that's
a
single
file
that
told
me
that
I've
created
the
table.
Okay,
so
now
I'm
gonna
go
ahead
and
run
a
string
right.
A
So
underneath
the
covers
I'm
gonna
go
back
to
how
Delta
Lake
works,
but
I'm
just
gonna
go
ahead
and
show
a
sort
of
quick
demo
of
what
I
mean
by
this
concept
of
unifying
streaming
and
batch.
Okay
and
so
I'm
gonna
run
this
quick
example
on
a
very
small
table
that
basically
shows
me
the
loans
button,
the
count
loans
by
state,
the
number
of
loans
you
have
in
each
particular
state,
okay
and
so
right
now
what
you'll
notice?
Okay,
I'm,
just
gonna,
run
this
real
quick
and
is
that
I've
got
a
read
stream.
A
I
eat
streaming
to
read
the
data
it
from
this
loan
by
read
stream
table.
This
loan
arrives
restream
tables
actually
pointing
to
this
Delta
Lake
silver
path,
which
I've
defined
early
on.
It's
basically
a
location.
The
reason
I'm
calling
out
this
path
is
because
it's
the
same
path
as
our
batch
table,
so,
whether
I'm
doing
this
vibes
streaming
or
via
bash,
it's
actually
going
to
the
same
file,
location,
okay
and
so
alright,
so
right
now
notice
that
Iowa's
null
and
watching
States,
336,
okay,
so
I'm
gonna
go
ahead
and
run
this
now.
A
A
Actually,
it
doesn't
fail
because
I
actually
already
fixed
the
code.
My
bad
again
running
live
streams,
so
alright
well
right
now
works
perfectly.
Fine.
I
had
actually
left
this
originally,
where
I
forgot
to
head
convert
this
into
an
integer
too
long,
and
so
because
it
did,
it
actually
would
break
down.
This
is
the
concept
of
schema
forcement,
where
basically,
the
Delta
Lake
will
prevent
incorrect
schema
data
from
being
inserted.
Now,
of
course,
I
went
and
fixed
it
already.
A
So
now
data
is
going
in
so
right
now,
I've
got
this
stream
of
data
going
and
how
does
the
process
rate
it's
relatively
slow,
but
the
context
is
that
now
Washington
State,
which
I'll,
which
was
written
in
owl
now
as
5:40
Washington,
which
was
originally
336,
now,
is
876
and
that's
pretty
cool,
so
I'm
actually
able
to
read
the
stream
and
rate
the
stream
that's
exact
same
time.
What's
also
cool
is
that
I
can
actually
go
ahead
and
write
yet
another
stream
into
this
data.
So
this
is
my
third
second
stream.
A
So
I've
got
to
writing
streams
writing
to
the
exact
location
while
I'm
also
reading
a
stream
from
it
at
the
exact
same
time,
and
so
you
notice
that
Iowa
and
Washington
I'm
jumping
up
alright.
So
this
is
pretty
cool,
because
if
you
look
at
a
regular
park
a
table
or
something
of
that
nature
be
able
to
go
ahead
and
read
and
write
from
this
same
table
at
the
same
time
is
actually
pretty
tricky.
If
not,
if
not
break
it
all
breaks
down,
and
for
that
matter
now,
I'm
gonna
run
a
batch
query
against
that
data.
A
Okay,
so
select
now
you
notice,
it
says:
Bailon
like
State
Delta,
as
opposed
to
the
low
by
state,
read
stream,
mainly
because
I'm
actually
reading
the
batch
data,
not
actually
the
the
stream
data
in
your,
and
so
it's
steadily
increasing
over
time
and
so
right
right.
Originally,
my
batch
was
at
a
20
74
Iowa
I
go
back
up,
it's
actually
already
at
twenty
eight,
eighty
okay,
so
because
the
stream
is
running
faster.
So
if
I
run
a
yet
another
batch
inside
here,
I
probably
will
catch
up
too
much.
So,
let's
see
Wow,
okay.
A
But
the
context
basically
is
that
I
can
write
my
streams,
read
stream,
do
to
write
streams
and
a
batch
all
the
same
data
all
in
the
same
location,
there's
still
two
like
silver
path,
all
concurrently,
alright-
and
we
can
do
this
under
the
cover,
because
if
you
look
at
the
Delta
logic
here,
alright
you'll
notice
that
actually
underneath
the
covers
here
I
have
all
these
different
JSON
files
for
this.
So
this
is
a
transaction
for
each
and
every
one
of
these
things
to
basically
go
showcase.
The
fact
that
okay
I
have
a
transaction.
A
Every
insertion
by
every
write
stream
has
a
transaction
that
protects
the
data.
Alright
I'm,
going
to
talk
about
this
checkpoint
in
a
second.
But
the
context
here
is
that
every
since
the
two
streams
that
we're
doing
the
writes
are
actually
going
ahead
and
now
showing
up
as
transactions
inside
here.
In
fact,
if
I
was
to
go
ahead
and
go
back
and
run
that
describe
history,
I'm
just
gonna
grab
that
watch.
You
know
what
I'm
gonna
jump
to
that.
A
little
part.
A
little
bit
later,
I'm
just
gonna,
look
at
the
files
for
now.
A
So
let
me
go
do
that
if
you
look
at,
for
example,
two
different
files
like
14
and
15
14,
13
you'll
notice
that
basically
the
timestamp
is
in
here
and
then
who
is
the
user?
Id
I
was
the
one
who
did
it,
what
the
operation
was
and
so
and
then
all
that
was
inside,
that
transaction,
okay
and
all
the
values
that
were
modified
as
part
of
that
transaction.
Okay,
so
I
was
doing
these
changes
to
Washington,
State
and
I
was
thinking
the
changes
to
Iowa
State
right
see
my
deal
with
the
previous
one.
A
Thirteen
right,
it's
a
different
time
stamp.
Okay,
it's
at
6:59
here
versus
900
in
terms
of
back
to
the
epochs,
but
same
idea,
Oh
what
happens
to
Washington
State
and
what
happens
to
Iowa
State
they're
all
side
here.
Okay,
now
this
context
is
important
because
that's
what
we're
trying
to
we're
trying
to
implement
atomicity,
okay,
so
changes
to
the
table
are
stored,
as
ordered
atomic
units
called
commits
right
near
you
commit
a
transaction
well,
so
exactly
as
you
saw
in
the
JSON
files
when
we're
running
this
demo,
alright,
you
add
a
parquet.
A
You
first
client
tries
to
add
a
park,
a
second
client
as
tries
to
add
a
parquet,
or
they
try
to
remove
a
parquet
files.
It
doesn't
matter,
we
record
all
those
transactions,
everything
that
occurs
the
changes
to
the
data,
the
changes
to
the
files
they
show
up
in
the
stock,
JSON
that
you
just
saw
right
so
that
way.
Now
we
have
a
log
that
records
everything
that's
happening
and
because
of
that,
we
can
allow
multiple
streams
to
be
writing
to
the
same
table.
A
Concurrently,
that's
why
you
could
go
ahead
and
have
and
don't
like
I
could
go
ahead
and
actually
have
those
two
streams
right
into
the
same
location
and
the
batch
query,
reading
from
the
same
location
and
the
read
stream
of
reading
from
same
location
without
any
problems,
alright,
but
how?
If
they're
conflicts?
Okay,
so
if
you
remember
the
the
the
notebook
I
know,
it's
the
time
stamp
write
this.
So
it's
really.
Basically,
we
have
a
time
stamp
that
tracks
everything.
Okay,
so
we
record
that
start
person.
A
We
record
the
reads
and
writes
we
attempt
to
commit
if
someone
else
wins
check.
If
you
have
anything
to
read
anything,
you
are
trying
to
do
house
that
has
changed
and
try
again,
okay,
so
in
other
words,
if
it's
just
like
adding
data
like
what
we
did,
it's
relatively
straightforward
because,
right
even
if
for
sake,
argument,
two
of
those
transactions
were
to
run
exactly
the
same
time
and
they
you
know
they
landed
it
on
the
exact
epoch.
What
would
happen
is
basically,
let's
go
right
to
the
disk.
A
First
client
wins:
okay,
it's
written
second
client.
It
goes
ahead
and
realizes
there's
a
problem.
Now
we
could
error
out,
but
because
we're
trying
to
resolve
this
optimistically,
what
we're
doing
is
we're
actually
going
ahead
and
running.
It
will
retry
because
we're
saying,
hey,
read
the
data
the
what
I'm
about
to
do
this
insertion
this
append,
isn't
gonna
Maksim,
make
any
impact
right
verses,
an
update
or
delete
right,
which
it
could
have
an
impact.
So
in
that
case,
go
ahead
and
just
write
optimistically
go
ahead
and
write
this
and
ie
try
again,
okay!
A
Now,
if
it
can't
do
it
by
the
way
it
will
error
out.
So
no
words
if
you're
say
Cartman
somebody's
trying
to
do
a
delete
and
update
at
the
exact
same
time,
then
what
ends
up
happening
is
that
sure
one
of
them
will
win
the
second
one.
That's
trying
to,
let's
just
say:
do
the
delete
will
assert.
You
see
me,
do
the
update
when
the
first
one
I
was
doing
a
delete.
A
The
second
one
will
then
fail
and
roll
back
to
the
client
that
let
them
know
hey,
you
just
try
to
run
an
update
on
a
non-existent
file
or
non-existent
set
of
data
all
right.
So
how
do
we
also
approach
this
all
this
day,
because
what
this
implies
is
that
we're
adding
more
and
more
data
right
to
the
system?
You
have
add
all
these
park'
files
right,
and
so
these
large,
just
like
we
were
saying
the
large
tables,
have
millions
of
thousands.
So
how
do
we
scale
the
metadata?
So
we
actually
use
spark
to
scale
this.
A
So
what
what?
How
actually
happens
like
so
each
of
these
Jason's
comes
in,
we
add
/,
ki
deprecated
remove
remove
whatever
it
is.
They
have
these
transactions
and
after
every
10
of
them,
we'll
do
a
check
point
into
park',
and
so
then
that
way,
when
we
need
to
read
the
table,
we're
gonna
first
read
the
transaction
log,
but
instead
of
trying
to
read
every
single
JSON,
that's
inside
there
we're
gonna
read
the
park'
files
of
our
transaction
log.
The
check
points
put
that
into
memory.
A
So
that
way,
it
only
actually
has
to
read
the
new
JSON
files,
okay,
so
the
new
transactions,
and
then
we
can
keep
easily
keep
track
of
all
the
files
that
have
been
changed
in
the
system.
So,
let's
go
back
to
the
demo
real,
quick,
you'll
notice,
if
I
scroll
back
up
on
the
tenth
one,
it
says
checkpoint,
parquet
right
and
then
on
to
20th
one
same
idea
at
the
checkpoint.
Parquet,
that's
exactly
what's
happening
here
right!
This
goes
back
to
that
scaling.
Metadata
right!
We've
checked
pointed
it.
A
If
you
shut
down
the
cluster
or
bring
up
a
new
cluster,
and
you
need
to
scan
and
understand,
what's
going
on
again,
it'll
just
simply
go
ahead
and
read
all
the
Park
A's
right
away
so
put
that
in
memory,
and
so
it'll
only
have
to
read
21st
and
the
22nd
files
to
figure
out.
What's
going
on,
okay,
all
right!
So
that's
actually
how
Delta
Lake
works
under
the
covers
all
right,
and
so
the
L
Delta
architecture
right,
let's
go
into
that
specifically!
So
how
do
you
connect
all
these
dots
together?
A
A
But
how
do
I
put
this
together
to
simplify
my
flow
right?
How
do
I
connect
these
dots
together,
so
the
ability
to
read
consistent
data,
while
it's
being
written,
remember
so
remember
this-
that
quick
demo
I
wrote
tattoos
one
read
stream
to
write
streams
and
one
read
batch
right:
well,
because
there's
a
stash
left
isolation
between
writers
and
readers,
I'm
actually
able
to
do
that
now,
right,
I
can
I
can
that's
the
demo
you
saw.
A
You
are
able
to
see
us
going
ahead
and
doing
reads
and
writes
whether
a
stream
or
a
batch,
it
didn't
matter
all
right.
You
able
to
read
incrementally
from
a
large
table
through
with
good
and
throughput
okay.
So
how
do
I
do
that
because
we
had
optimized
file
source
with
scaleable
metadata
handling
and
that
when
you
saw
the
JSON
and
the
checkpoint
parquet,
that's
exactly
the
point
right
that
those
checkpoints
allow
us
to
basically
ensure
that
the
metadata
can
scale
as
well.
It's
bad.
A
It
wouldn't
be
good
if
we
could
scale
the
data
itself,
but
did
a
horrible
job
with
the
metadata
right.
So
that's
why
we
do
the
checkpoints
to
make
sure
it's
easier
for
us
to
scan
the
metadata
as
well.
So
that
way,
it
knows
exactly
what
files
it's
supposed
to
read.
We
also
have
the
ability
to
rollback
in
case
of
bad
rights,
in
this
case
time
travel.
So,
what's
time
travel
well,
let's
go
back
to
demos
that
alright,
so
I'm
just
gonna
skip
back.
A
We
actually
have
this
notebook
by
the
way
which
we
will
share
actually
has
a
lot
of
cool
other
examples,
but
I
just
wanted
to
show
you
that
the
main
things
here
right
now.
So
let's
go
back
and
look
at
this
table
all
right.
So
you
before
you
can
look
at
the
JSON
files,
but
I'm
just
gonna
run
it
in
terms
of
so
you
can
see
it
this
way.
Okay,
so
remember
how
the
JSON
file
there
are
22
of
them.
A
Well,
this
is
just
the
history
table
that
shows
you,
the
initial
creation,
0,
okay
and
then
also
all
the
22
different
versions
of
that
table
as
we
loaded
the
data
up.
Ok,
all
right,
so,
what's
great
about
time!
Travel
basically
is
this,
so
if
I
want
to
go
back
and
look
at
what
the
look
at
the
loan
bike,
State
Delta
table,
as
of
the
zeroth
version,
all
I
have
to
do
is
specify
version
as
of
0,
and
what
you
have
here
now
is
Iowa
back
to
a
null
value.
A
Okay,
so
I
was
null
and
Washington's
336.
That
was
the
version
of
the
table
when
it
was
originally
created
right
from
the
get-go
all
right,
and
then
we
remember
how
we
did
what
we
did
is
we
added
two
right
streams
to
load
up
the
data,
so
I'm,
just
I'm,
just
gonna
choose
none
number
nine,
okay,
and
so
this
is
a
ninth
version
of
that
of
the
22
different
versions.
A
We
have
ok
for
this,
for
example,
and
by
the
ninth
version,
Iowa
was
at
12:15,
Washington,
1821
and
California
is
still
a
to
the
2006,
as
of
as
of
version
9
as
a
version
0
right,
because
there
was
no
difference.
In
fact,
every
other
state
didn't
change,
there's
only
Iowa
and
Washington
State
and
then,
but
don't
worry.
A
If
we
want
to
get
the
original
data
back,
you
can
either
tell
specify
the
version
as
the
curtain
version
or
don't
even
bother
specifying
the
version
all
and
here's
the
current
version
or
Iowa's
now
at
3:00,
9:15
and
Washington's
at
4:00,
9
71
all
right.
So
this
concept
of
time,
travel
basically
is
allows
us
to
go
ahead
and
go
back
in
time
and
find
out
exactly
which
version
we
want
to
go
to
and
make
changes
as
you
steam
fit.
A
It
could
be
something
as
cool
as
just
showing
an
animation
like
this,
where
you
simply
seeing
what
the
different
version
looks
like
in
a
previous
session.
Getting
ready
data
ready
for
data
science
with
Delta
like
an
ml
flow
I,
showed
this
for
a
version
of
this
notebook
that
allowed
us
to
identify
model
drift
or
data
drift.
Where,
as
data
changes
over
time,
that
we
could
go
ahead
and
copy,
we
could
actually
monitor
that
and
see
the
impact
of
that
on
our
machine
learning
level.
The
same
comes
into
play
right.
A
It
could
be
something
as
simple
as
that
or
something
as
complex
as
running
rollbacks,
which
is
you
realize,
hey
the
last
10
additions
were
our
updates.
This
we're
actually
incorrect.
Let's
roll
back
to
a
previous
version,
let's
roll
back
for
sake,
argument
of
version
9,
so
no
problem
at
all.
We
can
go
back
and
do
that
right,
because
the
data
it
still
is
there
all
right.
So,
let's
continue
I
just
lied
all
right,
so
the
fourth
back
connecting
the
dots
again,
the
ability
to
replay
historical
data
along
new
data
that
arrived
right.
A
Well,
then
stream,
the
backfilled
historical
data
through
the
same
pipeline,
so
in
other
words,
if
you've
got
a
source
that
contains
all
the
old
data
that
would
was
delayed.
Not
a
big
deal
just
go
ahead
stream
it
through
this
exact
same
pipeline
or
if
you've
got
another
source,
or
you
need
to
replay
it
all
over
again,
because
the
business
logic
has
changed
again.
You've
got
the
full
source.
A
It's
basically
screamed
right
back
to
the
left
and
have
that
run
this
exact
same
process
all
over
again
and
the
ability
to
handle
late-arriving
data
without
having
to
delay
downstream
processing
and,
in
this
case
straight
forward
stream,
any
late
arriving
data
added
to
the
table
as
they
get
right,
because
now
you
simplified
it
so.
The
context
here
is
now
the
Delta
architecture
with
Delta
Lake,
as
allows
you
to
continuously
create
a
continuous
data
flow
to
unify
your
batch
and
streaming
again,
this
the
demo
was
relatively
simplistic.
A
Typically,
it
looks
something
closer
to
this,
where
you
have
multiple
sources
resulting
in
multiple
tables
or
multiple
data
stores,
even
right
and
the
this
concept
of
that
we
have.
You
see:
here's
bronze
silver
gold
of
tables
in
between
it's
all,
because
you're
trying
to
merge
different
tables,
different
data
sources,
together
with
different
levels
of
data
quality
right.
A
So
it
seems
like
it's
more
complicated
by
doing
this,
but
the
whole
premise
is
saying
that
I
can
now
actually
have
a
single
pipeline
back
to
this
thing
right
to
do
all
of
these
things
all
right,
and
so
that's
how
we
connect
the
dots
together.
So
now
that
I've
talked
about
this,
let's
talk
a
little
bit
about
the
characteristics
of
a
data
architecture.
A
Oh
actually,
I'm
gonna
jump
real,
quick
into
a
question,
because
just
because
there's
some
good
questions
before
I
talk
about
Dell
doctor,
let
me
actually
answer
some
questions
here.
Okay,
what
causes
a
version
update
is
a
related
to
checkpoint
the
check,
let's
break
those
two
things
down,
because
there
are
actually
different
things.
The
checkpoint
itself
is
every
10th
version
or
every
tenth
transaction
to
that
table.
We're
gonna
do
a
checkpoint
so
that
way
it
creates
a
park.
A
file
that
park
a
file
then
gets
loaded
into
memory
right.
A
So
this
way,
instead
of
actually
trying
to
read
all
these
small
JSON
files,
because
you
know
just
like
any
other
SPARC
infrastructure
or
any
infrastructure
right,
the
more
small
files
you
have
to
read
the
larger
the
overhead
to
do
it
right.
So,
instead
of
actually
doing
that
to
you,
what
we
do
is
we
create
these
little
park'
files
and
we
put
them
into
memory.
So
if
it's
an
existing
process,
existing
cluster
spark
cluster,
that's
running
no
problem,
it's
already
floating
a
memory
and
then
only
the
new
JSON.
A
The
new
transactions
actually
have
to
record
it
right
and
processed
and
then
by
the
10th
one
again
we
checkpoint
it
and
we
took
it
back
the
memory.
If
you
shut
down
the
cluster
and
you
start
a
new
cluster
to
access
that
table,
the
parkade
files
are
there.
So,
instead
of
actually
reading
all
the
JSON
files,
we'll
just
read
all
the
Parque
Fox
and
so
can
chuck,
it
can
actually
chuck
that
stuff
into
memory
faster.
So
it
exactly
knows
exactly
what
transaction
that
actually
happened.
Ok,
so
what
causes
a
version
update?
A
Well,
the
version
update
like
is
actually
a
transaction,
so
in
other
words,
any
changes
that
you
do
to
the
table,
so
an
insert
or
an
update
or
delete
or
merge.
That's
actually
how
you
would
physically
go.
Do
this?
Ok,
so
hopefully
that
answers
that
question
right
now
relate
to
that.
There's.
Another
question:
ok,
is
that,
is
it
possible
to
update
it?
A
How
do
I
delete
a
version
of
data?
Ok,
so
typically,
if
you
want
to
delete
something
you're
deleting
the
data
itself
as
opposed
to
deleting
a
specific
version
of
it,
ok
now
there's
this
concept.
The
reason
why
is
because,
from
a
gdpr
compliance
perspective,
you
want
to
be
able
record
every
single
transaction
that
occurs.
Okay,
that's
actually
the
context.
A
So
what
happens
is
that
if
I
want
to
go
ahead
and
remove
all
of
those
files,
though
right,
because
what
what
implant
what's
implied
very
strongly
here,
is
that
the
when
you
run
a
delete,
you
actually
don't
delete
it
right.
You
you've
kept
the
files.
That's
why
I'm
actually
able
to
see
the
zeroth
version,
the
ninth
version
of
the
table,
because
I
actually
still
have
the
old
files.
A
I
just
simply
have
a
transaction
log
that
tells
me
which
version
I'm
supposed
to
be
looking
at
which
sort
of
files
I
should
be
looking
at
when
I
have
when
I
request
for
an
older
version
of
that
data,
but
if
I
don't
go
ahead
and
actually,
if
I,
do
really
want
to
remove
it.
Well,
how
do
I
do
that?
That's
running
a
vacuum!
Okay,
so
by
running
a
vacuum
statement,
basically,
you
clear
out
the
files
by
default.
A
Any
files
are
physically,
so
there
are
some
days
older,
okay,
so
in
other
words,
what
happens
that
when
I
run
the
vacuum,
it
clears
out
the
data
once
it's
cleared
out,
then,
if
I
go
back
and
request
for
that
old
version
of
the
data,
it'll
give
you
an
error
message,
simply
saying
that
data
and
no
longer
exists,
but
you
still
have
a
transaction
log
that
denotes
all
the
actions
you
did
to
the
table
up
to
that
point.
But
yes,
the
data
no
longer
exists.
A
Okay,
if
you
need
that
data
to
be
removed
right
away,
then,
instead
of
using
default
some
days
just
specify,
for
example,
zero
right,
so
they'll
just
mean
give
rid
of
everything
right
from
the
get-go.
Okay,
all
right
perfect.
So
let
me
go
ahead
and
talk
about
the
Delta
architecture,
a
little
bit
more.
The
actual
purpose
of
this
session,
okay,
here
the
some
of
the
key
tenants
that
we
really
want
to
do
for
this
okay,
so
you
want
basically
want
to
adopt
a
continuous
data
flow
model.
A
Okay
and
what
we
mean
by
that
by
the
way
is
that
you
want
to
stream
to
and
from
the
Delta
lake
table
whenever
possible.
Okay,
so
in
other
words,
it's
not
just
about
going
ahead,
even
if
you
think
of
the
problem
as
a
batch
problem,
you
actually,
if
you
can
just
think
about
it
purely
from
a
standpoint
of
like
all
the
jobs
are
extreme
jobs
and
all
the
jobs
are
to
and
from
the
Delta
Lake.
A
It
allows
you
to
go
ahead
and
do
these
three
tests
you
can
unify
back
to
stream
same
engine,
same
API,
same
user
code,
the
no
need
to
reason
about
system,
complexity,
separable,
okay
and
then
you're
able
to
increment
a
load
the
new
data
very
efficiently.
Okay,
in
other
words,
you
know,
there's
not
there
isn't
a
need
to
do
steep
management.
Okay,
on
what
new
files
are
added.
Okay,
you
basically
incrementally
add
and
you're
good
to
go,
and
then
you
process
the
data
super
quickly
right.
So
there
are
limited
delays,
minimal
delays
to
note
it
likes.
A
Okay,
so
I
forgot
if
I
I
included
this
in
the
slide.
But
you
know
one
cool
example
that
we
had
from
last
year's
SPARC
summit
in
San
Francisco
when
the
keynote
speakers
was
Comcast
and
what's
really
cool
about.
The
Comcast
scenario
is
that
they
used
to
have
84
jobs.
Okay-
and
this
is
for
this
particular
example
of
their
system.
The
session
ization
is
basically
basically
with
the
84
jobs,
because
they're
able
to
use
Delta
like
and
because
they
had
to
have
such
actions
and
because
they
follow
the
Delta
searcher.
A
They
converted
those
84
back
jobs
into
three
streaming
jobs
right
so
from
a
maintenance
and
operational
perspective,
you've
simplified
the
workload
massively
okay,
all
right.
So,
let's
see
use
intermediate
hops.
Okay,
materializes
data
frames
whenever
it's
applicable,
especially
when
a
large
number
transformations
evolve
right,
you're,
Mitchell
I
says
you
can
be
used
for
fault
recovery,
easy
troubleshooting
and
multiple
consumers
are
expected.
Alright.
So
what
I
mean
by
this?
Is
this
like
traditionally
right?
If
you've
got
a
source
table
and
you
get
your
destination
tables?
A
What
we'll
do
is
we'll
do
multiple
transformations,
okay,
and
so
these
intermediate
data
frames
that
you
create
are
typically
sitting
in
memory.
Okay
and
obviously
you
do
this
because
it's
faster
to
process
this
data
in
memory.
Okay,
what
we're
saying
is
that
if
you
actually
have
concerns
or
have
questions
or
want
to
be
able
to
ensure
that
the,
if
there's
a
breakdown
in
tax,
because
the
transformation
you
have
is
relatively
complicated
well,
what
asking
to
do
is
like
want
to
create
an
intermediate
table
right.
It
actually
simplified
some.
A
The
flow
for
example
in
this
example
tables
1,
2
&
3,
were
actually
useful
for
both
the
both
the
the
output
destination
table.
There's
just
from
Table
three
onwards.
We
needed
actually
to
go
ahead
and
to
do
something
slightly
different,
so,
instead
of
creating
two
separate
set
of
tables
for
with
the
exact
same
data,
go
ahead
and
actually
run
in
an
intermediate
table,
save
that
to
disk
and
then
run
two
separate
processes,
row
for
table
for
Table,
six
Tim,
five
Table
seven
are
a
different
stream
and
they
go
to
different
tables.
A
It
really
simplifies
the
flow
here
and
by
the
way
I
apologize,
but
for
some
reason
my
Q&A
panel
is
not
showing
up.
So,
in
fact,
I
want
to
ask
you
if
it's
okay
for
you
to
post
your
question
to
the
zoom
webinar
chat
I
realize
that's
actually
not
a
good
best
practice,
but
for
some
reason
my
Q&A
panel
is
not
showing
up
right
now,
so
please
put
them
in
the
zoom.
Webinar
chat,
okay,
and
sometimes
because
you
have
an
issue
where
you're
you
don't
know.
A
If
the
data
is
partitioned
correctly
or
you
running
into
SKUs
or
whatever
else
yeah,
you
put
the
intermediate
table
even
earlier
up
and
that's
perfectly
fine,
okay,
so
the
context
again
for
most
and
learner
fault
recovery,
multiple
consumers
are
expected
easy
troubleshooting
boom,
go
ahead
and
actually
use
the
intermediate
hops.
Okay,
all
right,
and
so
now
that
we've
set
all
this.
What's
the
cost
versus
latency
trade-off?
A
Okay,
so,
for
example,
if
you've
got
streams
of
data
arriving
continuously,
what
what's
implied
is
that
you
have
an
always-on
cluster
right
and
you're
continuously
processing
data
to
do
that
that
that
in
itself
seems
like
a
more
costly,
potentially
right.
So
how
do
you
get
around
that
right?
There
there's
there's
examples
of
example:
you
run
frequent
batches.
You
data
is
arriving
every
few
minutes
or
even
thirty.
A
Thirty
minutes,
whatever
use
a
warm
pool
of
machines,
so
then
turn
off
the
cluster
when
it's
idle
and
start
the
cluster,
when
the
data
needs
to
be
processed
and
then
you,
for
example,
using
the
streaming
trigger
not
once
mode
so
the
cluster
will
just
kick
off
and
you're
good
to
go
right
or
you've
got
infrequent
batches.
You
get
data
arriving
every
few
hours
a
day
is
turn
off
the
cluster
idle
start
the
cluster.
When
the
day
needs
you
processed
again
used
streaming
triggered
at
once
mode.
Ok,
the
the
context.
A
Basically,
then,
potentially,
you
can
go
ahead
and
actually
run
the
clusters
or
kick
off
the
clusters
only
when
you
physically
need
them,
ok
to
make
sense
of
it
now.
One
thing
to
note,
for
example,
is
that
when
Comcast
back
to
that
example,
when
a
hand
switched
over
because
of
the
efficiencies
for
them
to
run
this
continuously
with
three
jobs
and
so
to
e4,
they
went
ahead
and
shrunk
down
from
640
VMs
to
64
a
10x
improvement,
ie
reduction
of
the
number
of
virtual
machines,
so
the
cost
of
them
doing
this.
A
A
Alright,
another
important
aspect
of
the
Delta
Carter
architecture
is
also
be
able
to
reprocess
the
stable
right,
so
infinite
retention
of
raw
data,
Plus
stream
is
that
means
you've
got
a
relatively
trivial
recomputation
of
that
data.
Okay
and
oh,
by
the
way,
the
question
was
asked
like
what
is
trigger
got
once
this
is
part
of
structure
streaming,
so
we
actually
will
have
a
separate
session
to
dive
deeper
into
structure
streaming.
They'll
be
coming
up
in
the
next
few
weeks.
A
In
fact,
so
we'll
talk
about
that
then,
but
the
long,
the
idea,
the
context
is
basically
within
structure
streaming.
Once
you
run
the
trigger.
Basically,
it
basically
tells
structure
streaming.
Okay
data
is
in
trigger
it
to
run
it,
and
we
can
also
tell
the
cluster
go
ahead
and
kick
itself
up
to
start
processing
that
data,
ok,
and
so
what
we
mean
by
this
reprocessing
perspective
is
that
you
can
simply
clear
out
the
result,
table
and
restart
the
stream
all
over
again.
Okay.
A
So,
in
a
previous
session,
we
often
we'd
over
in
this
concept
of
bronze
silver
gold
tables
for
data
quality,
of
the
ideas
that
the
bronze
is
your
raw
data
dump
it
you
just
dump
the
data
and
you
put
it
inside
there.
The
silver
is
this
concept
of
filter
and
augmented
and
cleanse
data.
So
as
you
progress
in
from
bronze
and
to
silver,
the
data
is
a
lot
cleaner
and
then
you
from
silver
to
gold,
and
you
do
the
final
bit
of
augmentation
or
cleansing.
A
So
that
way
you
can
run
your
bi
or
AI
or
data
science.
Your
machine
learning
against
that
data.
These
are
all
really
good
things,
but
the
context
is
that
how
about
if
the
business
logic
for
the
augmentation
or
cleansing
was
incorrect
or
needs
to
be
updated?
Well,
that's!
What's
great
about
you
still
have
the
bronze
table
that
has
all
the
raw
data
from
the
first
place,
go
ahead
and
reprocess
that
all
over
again,
okay
and
then
you
can
recreate
the
silver
or
create
the
gold
and
you're
you're
you're
good
to
go
right,
leverage
cloud
elasticity.
A
So,
in
other
words,
because
you
have
to
go
back
in
time
and
reprocess
all
the
data
scale
up
right,
add
more
notes
purely
for
the
purpose
of
getting
that
backfill
figured
out
once
that
back
that
will
figure
it
out
boom
flip
the
switch
you
can
scale
back
down
and
to
your
regular
amounts
of
processing,
okay
and
and
this
actually
there
we
go
tune.
The
data
quality
right.
So
this
is
what
I
I
was
just
referring
to
us.
A
So
I
forgot
that
I've
done
this,
but
basically
here
in
the
bottom
left,
here's
your
streaming
and
your
back
sources
as
you're
progressing
from
left
to
right
from
broad
and
silver-gold.
This
allows
you
to
improve
that
data
quality
such
that
your,
whether
it's
streaming
analytics
or
your
AI
reporting
you're
good
to
go
in
the
process
of
doing
this.
You
want
to
go
ahead
and
ensure
to
force,
but
also
be
able
to
merge
the
schemas
okay.
So
there
are
times
where
you
want
to
enforce
the
schema.
A
A
You
may
want
to
merge
the
schema
right
where
you
basically
want
to
add
new
columns
as
they
come
in,
but
what
you
want
to
do
is
you
want
to
make
sure
that
you,
when
you
allow
the
schema
to
evolve,
it
doesn't
matter
really
in
which
level
right
when
you
do
want
it
to
devolve
you
want
to
make
sure
that
it
actually
is
what
you
want.
So
by
definition,
Delta
Lake
enforces
a
schema,
and
then
you
have
to
specify
that
you
want
to
evolve
the
schema.
A
So
that
way
you
know
you're
actually
going
and
making
changes
to
the
schema
right.
You
can't
you
don't
want
to
do
this
as
a
haphazard
function.
Okay,
all
right
so,
like
I
noted
before
here's
a
summary
of
those
key
characteristics
right,
adopt
a
community
or
state
of
flow
use,
intermediate
hops
to
improve
the
reliability
and
troubleshooting
and
make
the
cost
first
relating
latency
trade-off,
based
on
your
use
case
in
business
needs
and
optimize.
A
The
storage
layout
based
on
your
access
patterns,
we
process
the
historical
data
as
you
need
and
increment
to
improve
the
quality
of
your
data
for
our.
So
it's
ready
for
consumption.
Okay.
This
is
these
are
the
key
characteristics
of
what
you
have
with
the
delt
architecture
that
will
base
it
will
allow
you
to
simplify
your
flow.
Okay.
The
benefits
of
this
architecture
then
reduces
the
end
to
end
pipeline
SLA.
A
A
merged
statements
directly
against
your
data,
using
either
the
Python
Scala
API,
is
currently
or
once
SPARC
3.0
comes
out
with
data
for
speak
to
with
Delta
Lake
you're
gonna
be
able
to
run
that
and
see
what
sequel,
Stevens
as
well
also
be
able
to
do
deduplication,
and
then
you
know:
lower
infrastructure
cost
to
the
last
independent
computed
storage.
Right
right,
like
said
we,
we
have
organizations,
reduced
infrastructure
cost
by
up
to
10x,
and
that's
exactly
what
Comcast
had
done
again.
A
So
one
thing
I
wanted
to
finish
off
with
before
I
go
dive
into
the
rest
of
the
questions
is
that
you
know
as
more
and
more
people
are
starting
to
use
Delta
Lake.
In
fact,
we
have.
There
are
more
more
connectors,
so
Delta
Lake.
Can
you
talk
to
hive?
Obviously
spark
presto,
redshift,
Athena
snowflake
right
and
that's
the
current
as
of
today
this
this
is
rapidly
gonna
change,
ie,
adding
more
to
the
list.
Okay
hive
is
currently
in
I
believe
there's
a
question
that
came
up
on
the
chat
hive
is
currently
in
private
preview.
A
Now
we're
working
with
the
community
to
make
this
available
so
that
way,
Delta
like
can
go
ahead
and
work
on
hive
in
for
public
preview.
Obviously,
okay,
there's
a
lot
and
provider
so
whether
it's
your
tableaus
or
from
attica's
clickstream
sense
town,
the
a
toonie.
When
does
scope
for
this
era,
you
know
that
actually
work
closely
with
us
recently
was
announced
by
Google
data
proc
that
they
also
will
be
readin,
nothing
like
themselves
as
well.
A
So
this
list
is
gonna
rapidly
grow
as
well,
and
so
the
ecosystem
is
growing
as
more
and
more
people
and
companies
are
recognizing
the
benefits
of
actually
having
transaction
protections
around
their
distributed.
Storage,
sisters,
okay
and
let's
see
okay
and
plenty
users,
don't
feel
like
actually
just
go
to
Delta
diode.
You
can
see
the
full
list,
so
this
is
just
a
partial
list
of
it,
but
there's
a
lot
of
companies
out
there
that
are
all
standardizing
on
Delta
Lake
in
order
for
them
to
run.
A
In
fact,
a
data
breaks
itself
actually
more
than
75%
to
exabytes
of
data
per
month
or
actually
processed
using
Delta
Lake.
Now
so
it's
a
it's.
A
pretty
well
used
well
a
productionize
project
at
this
point.
So
if
you
do
have
any
questions,
please
join
us
at
Delta
IO.
We
actually
have
a
pretty
active
community,
whether
using
github
or
slack
or
whatever
else,
we're
pretty
much
talking
all
the
time
answering
questions
so
if,
in
case
I'm
not
answering
the
questions
now,
you
can
definitely
chime
in
there
and
I
did
want
to
leave.
A
Also
I'd
love
for
you
to
join
us
at
spark
summit,
spark
a
NATO
summit
2020
in
San
Francisco
as
well
this
year
from
June
22nd
to
25th,
and
so,
if
you
want
to,
please
use
the
discount
code.
Denne
denne
si
I
0:24
a
20%
discount
for
coming
to
spark
summit.
Ok,
so
with
that
I'm
gonna
leave
about
five
minutes
to
answer
any
questions.
Please
continue
putting
your
questions
inside
the
webinar
chat,
not
in
the
QA.
A
A
So
a
great
question
for
first
starters:
what
are
the
performance
implications
for
storing
historical
data
forever?
You
know
whether
you've,
slowly
changing
or
whether
you're
querying
newest
Oracle
data,
the
performance
implications
for
storing
the
data
forever
is
pretty
much
the
same
as
what
you
would
expect,
as
when
you
have
too
many
small
files-
and
you
have
many
many
many
many
of
them
write
it
over
time.
It
gets
larger
and
progressively
gets
slower.
A
The
only
way
to
speed
things
up
is
to
go
ahead
and
actually
put
more
resources
to
compute
against
it,
not
really
the
optimal
thing.
So
then
I
just
talked
about
time
travel
and
you
said
hey,
but
I
can
do
time.
Travel
and
I
can
go
ahead
and
go
back
and
figure
out.
What's
going
on
and
I
can
keep
a
circle
date
if
I
want
to.
Yes,
you
can,
but
those
are,
as
you
can
now
tell
they're
sort
of
slightly
competing
with
each
other.
A
So
obviously
every
practice
is
a
general
thumb
and
can
easily
be
broken,
but
the
general
rule
of
thumb
is
you
keep
the
data
for
about
three
months
right
long
enough
for
you
to
avoid
the
oops
issues
right,
in
other
words,
I
accidentally
deleted
this
or
I
accidentally
update
or
accidentally
inserted.
This
data
right
long
enough
for
the
oops
scenarios
to
be
solved
long
enough
for
you
to,
for
sake
of
argument,
take
care
of
looking
at
model
drift.
A
Our
data
drift
as
data
changes
over
time
to
see
how
that's
impacting
your
machine
learning
model
without
actually
having
a
massive
cost
to
the
system
on
holding
all
of
that
data,
and
then
invariably
the
question
will
be
hey
how
about
if
I
want
to
keep
older
versions
of
the
data?
Well
then,
in
your
case
at
least
right
now,
what
we
would
suggest
is
go
ahead,
then
save
that
version
of
the
table,
since
you
can
go,
for
example,
select
star
from
loan
by
States
Delta
we're
version
as
a
version
9
right.
A
If
you
need
that
version
9
table
what
I
do
is
just
simply
query
that
table
and
then
create
a
new
data
frame
or
sorry.
A
new
table
store
that
table
as
a
separate
source
write
a
separate
location.
So
that
way,
you
can
always
refer
to
that
version
of
the
table
and
put
it
in
archival
store.
So
that
way,
it's
not
like
you're
you're,
not
like
frequently
going
to
get
it,
but
for
segment.
A
A
So
that
way
you
can
still
go
back
in
time
get
that
version
of
the
data
like
I,
said
that
financial
month
end
right
without
actually
going
ahead
and
adding
to
the
burden
of
actually
trying
to
query
every
single
version
of
data
when
you
realistically
only
need
like
twelve
of
the
year
okay.
So
hopefully
that
answers
your
question
on
that
one.
A
So
another
really
good
question
is:
how
can
I
handle
dynamic
schemas
on
Delta
tables?
Can
Delta
theirs
create
new
columns
dynamically,
depending
on
the
schema
it
potentially
can
okay,
so
there
is
a
option
called
not
option.
Merge
schema
equals
true,
and
that
basically
means
it
implies
what
it
sounds
like,
which
is
when
I
specify
that
that
means
I'm
telling
Delta
like
you,
can
go
ahead
and
take
merger
scheme
and
say:
hey
I've
got
a
column
blah
that
was
not
in
this
original
table.
Let
me
go
bird
sat
together
now.
A
The
key
context,
basically,
is
that
you're
able
to
go
ahead
and
by
default,
enforce
the
schema
to
prevent
users
from
accidentally,
overwriting
or
accidentally
corrupting
the
table
right
that
that's
by
default.
We
want
to
prevent
that
from
happening,
but
then,
if
you
know
you
need
to
make
the
change
then
biospecimen
option.
Merge
schema
is
true,
you're
able
to
go
ahead
and
make
that
change,
and
so
you
use
that
concepts
very
sparingly
as
opposed
to
run
that
all
the
time
just
because
you
don't
want
people
to
change
the
schema
on
your
table
all
the
time.
Okay.
A
Okay,
I'm
gonna
answer
to
other
questions
and
at
the
end,
today's
session,
but
all
right
so
there's
a
question
here:
it's
like,
as
the
best
practice,
should
I
keep
versions
day
only
for
small
data,
or
is
it
a
good
practice
for
huge
large
huge
data
and
keep
those
versions
as
well,
and
so
the
good
practice
actually
is
not
so
much
the
size
of
the
data.
It's
more.
Your
business
use
case
right
for
firm
financial
information.
A
I
definitely
have
seen
it
common
where
I
will
go
ahead
and
save
like
I
said:
I'll
keep
the
versioning
on
my
Delta
like
table
on
for
three
months
and
then
I'll
costly,
clean
it
out,
but
by
default
seven
days
and
for
some
people
that's
perfectly
fine.
Okay,
and
by
the
way
you
actually
have
to
run
the
vacuum.
It's
not
like
it
automatically
happens.
You
run
the
vacuum.
It
just
says:
okay,
well
anything
older
than
seven
days
go
ahead
and
remove
okay,
but
so
it's
perfectly
fine
for
you
to
go
with
that.
A
Often,
like
I
said
it's
more
like
closer
to
thirty
days
or
90
days,
due
to
D
for
debugging
purposes,
but
then
just
like
I
noted,
where
for
like
financial
month
ends
where
I
want
to
keep
know
exactly
what
the
financial
amount
was
for
that
table,
because
I'm
doing
producer
responsibility,
I'll
save
that
actually
into
a
separate
table
for
archival
purposes.
Only
and
then
the
idea
is
that
if
I
have
any
issues,
I
can
go
back
and
look
at
okay.
Well,
what
was
we
were?
A
What
we
reported
in
the
month
of
of
January,
because
our
finding
all
our
financial
reports
generated
for
the
month
of
January
are
based
off
of
this
version
of
the
table
and
same
thing
for
February,
March
and
so
forth
and
so
forth.
Right,
so
that's
the
context,
so
the
the
the
best
practice
concerning
archival
or
keeping
all
these
different
versions
is
really
dependent
on
your
actual
business
use
case.
Okay.
So
hopefully
that
answers
that
question
and
the
final
question
that
I
want
to
answer
is
that
there
is
a
question
concerning
training
stages
schedules.
A
Training
sessions
are
scheduled
from
22nd
to
25th
of
June
and
those
were
posted.
Yesterday
are
those
training
sessions
going
to
be
recorded
available
later
in
general?
The
training
sessions
for
SPARC,
some
of
them
sell,
are
not
made
available
for
for
view
that
they're
meant
for
people
who
actually
are
attend
the
conference,
because
it's
actually
hands-on
sessions
where
you
get
to
come
in
get
to
work
on
the
sessions.
We
have
TAS
and
experts
in
the
field
that
go
ahead
and
not
only
teach
the
course,
but
also
answer
all
the
questions.
A
So
the
context,
basically,
is
that
yeah
we
it's
hard
for
us
to
go
ahead
record
those
just
because
the
amount
of
time
and
effort
spent
for
not
just
the
attendees
for
coming
to
listen
and
learn,
but
also
from
the
people
who
are
teaching
all
the
stuff,
and
you
get
to
ask
questions
so
hopefully
that
answers
all
your
questions
for
today.
I
know
there's
some
other
ones,
but
I
do
I'm
cognizant
of
the
time
and
world
just
about
the
top
of
the
hour,
so
I
did
want
to
leave
it
with.
A
Thank
you
very
much
for
taking
the
time
to
attend
today's
session.
We
will
have
another
session
next
week
about
simplifying
scaling
data
engineering
pipelines
with
Delta
Lake,
so
I'd
love
to
have
you
join
that.
Meanwhile,
if
you
do
we'd
love
to
also
see
you
at
spark
summit,
so
please
join
us
and
here's
a
discount
code
again
denne
sa
i0
20
and
where
you
get
20%
off
I,
think
that
covers
it.
For
me,
Karen
and
yet.