►
From YouTube: Online Meetup | From Tahoe to Delta Lake
Description
Join Delta Lake committers Burak Yavuz, Shixiong (Ryan) Zhu, Tathagata Das, QP Hou, and R. Tyler Croy for a fun and informative “Ask Me Anything” session on the journey of Delta Lake.
One week before Data+AI Summit 2021 (https://databricks.com/dataaisummit/north-america-2021) this is your chance to ask questions from key contributors to the Delta Lake project especially on the how and why! Some come prepared with your questions! And yes, there will be a special appearance by Michael Armbrust as well! Databricks is proud to announce that Gartner has named us a Leader in both the 2021 Magic Quadrant for Cloud Database Management Systems and the 2021 Magic Quadrant for Data Science and Machine Learning Platforms. Download the reports here. https://databricks.com/databricks-named-leader-by-gartner
A
B
C
A
All
right,
thank
you
very
much.
Everybody
welcome
to
this
awesome
session,
in
which
we've
got
multiple
panels,
as
you
can
see
from
this
panel
view
that
we
have
today,
just
in
case
you're
wondering
we're
part
of
the
from
tahoe
to
delta
lake.
We
are
summercast
on
zoom
youtube
and
linkedin
all
at
the
same
time.
So
I'm
going
to
start
off
to
explain
what
the
session
is
and
then
we'll
do
a
quick
round
of
introductions
and
then
we're
going
to
dive
right
into
the
ama
so
be
prepared.
A
A
Right
there
we
go,
and
just
confirming
to
my
esteemed
colleagues
that
you
can
see
the
slides
correctly
looks
good,
all
right
well
again
from
tahoe
to
delta
lake
there's
a
reason
we
call
the
tahoe
delta
lake
you're
going
to
find
out
very
soon
shortly,
okay,
so,
but
before
we
dive
into
it,
I
want
to
call
out
there
are
always
upcoming
online
events
with
the
data
and
ai
beta.
You
can
check
us
out
at
meetup.com
data
ai
dash
online.
A
You
can
subscribe
to
us
at
our
databricks
youtube
channel
or
you
can
follow
us
on
the
databricks
linkedin.
So
please
we're
we're
a
lot
of
different
locations,
so
you
can
definitely
check
us
out.
The
next
thing
we
want
to
call
out
is
this
thursday
we
have
the
data
and
ai
online
meeting
of
community
lightning
talks
in
preparation
for
the
upcoming
data
and
ai
summit.
A
Next
week
we
have
four
awesome
speakers:
okay,
fraca
patana,
siobhan
g
sristaba,
alyssa,
vishnek
and
yan
zhang,
where
they're
gonna
be
talking
about
their
various
components
of
data
engineering
and
data
science,
so
really
cool
stuff
as
a
good
preparation
or
call
out
to
the
dated
ai
summit,
which
of
course
called
oh
of
course
I
skipped
the
slide.
My
apologies.
I
did
also
want
to
call
before
talking
about
the
dna
summon
data
brew
by
databricks.
A
It
is
a
seamless
plug
for
a
podcast
vidcast
series
for
myself
and
brooke
winig
this
last
season.
We
were
talking
a
lot
about
lake
houses,
which
also
featured
by
the
way
michael
armbrust
who's
here
today,
hi
and
also
a
I'll
call
out
to
mate
zaharia,
who
does
episode
one
for
mla
production.
We
actually
have
multiple
sessions
already
so
check
us
out
here,
go
to
data
data,
brew,
dot,
io
or
that
link
there,
and
you
can
listen
to
us
also
on
spotify
and
apple
podcasts.
A
Now
I
can
go
talk
about
data
in
ai
summit.
It's
next
week.
First,
two
days
is
training
three
days
of
keynotes
and
awesome
sessions,
200
plus
highly
technical
sessions,
and
we've
got
a
lot
of
cool
people
featured
here.
So
please
register
today,
we'd
love
to
see
you
there.
Most
of
us
that
you
are
hearing
on
today's
session
are
actually
gonna,
be
there
as
well,
so
definitely
be
on
the
lookout
for
us
all
right.
A
So
without
further
ado,
that's
it
for
the
last
slide,
so
we're
gonna
go
from
tahoe
to
delta
lake,
so
I'm
going
to
stop
presenting
and
we're
going
to
go
right
into
questions.
So
if
we
have
any
questions
put,
like,
I
said,
put
them
on
zoom
put
them
on
linkedin
or
put
them
on
youtube.
Okay,
the
kit
quit
the
kick
the
yeah.
Excuse
me.
The
key
call
out
that
I
want
to
say
that,
obviously
we
want
to
talk
about
things
that
are
related
to
delta
lake
and
tahoe.
A
D
A
E
A
A
B
A
Excellent
well,
then,
by
the
way
for
everybody
please
put
in
your
questions,
especially
if
you
have
any
for
michael
or
td,
just
because
they're
going
to
be
here
for
about
another
seven
to
eight
minutes
before
they
have
to
run
off
to
yet
another
pre-summit
event.
So
let
me
start
off
with
you,
michael
first,
since
you
know
obviously
boom.
How
did
you
even
conceive?
B
Yeah
great
question,
so
you
know
delta
really
started
with
structured
streaming,
so
at
the
kind
of
setting
the
context
me
ryan,
barack
td,
we're
all
on
a
team
at
databricks
called
the
stream
team,
and
we
were
kind
of
tasked
with
reimagining.
What
streaming
should
look
like
inside
spark?
We
knew
that
spark
streaming
was
pretty
popular,
but
it
also
had
a
bunch
of
limitations.
It
didn't
support
event
time.
You
couldn't
change
your
queries
and
recover.
There
were
some
scalability
problems
when
you
were
working
with
state,
and
so
you
know
our
mission
was
reimagine.
C
B
That
was
a
pretty
difficult
problem
to
build
a
system
that
was
guaranteeing
exactly
one
semantics
on
top
of
this
eventually
consistent
file
system,
and
so,
if
you
look
at
the
file
sync
inside
of
apache
spark
and
you
squint
it'll,
actually
look
quite
a
bit
like
delta.
It
has
this
idea
of
a
transaction
log.
It
tracks
files
that
are
valid
in
any
given
version.
It
even
has
nascent
unused
support
for
removing
files
that
have
been
added
to
the
transaction
log,
but
it.
B
B
C
B
Summit
and
I
met
somebody
who
wanted
to
use
structured
streaming
at
a
fortune,
10
company
to
ingest
trillions
of
records
per
day
and
petabytes
of
data
per
week,
and
I
thought
oh,
that
sounds
like
a
really
good
challenge.
But
there
is
no
way
the
open
source
file.
Sync
is
going
to
handle
that,
and
so
we
had
been
talking
for
a
couple
of
months
about
this
idea
of
a
scalable
transaction
log
and
what
it
would
be
like.
A
That's
really
cool,
so
then
this
is
more
open
to
anybody.
But
but
thank
you
very
much
for
that
context,
michael.
But
is
why
tahoe
what
what
was
the
code
name
tahoe
before
we
ever
call
this
delta
lake,
because
that,
by
the
way
for
everybody
to
provide
context,
the
reason
the
session's
called
from
tahoe
to
ducklink?
Because
that's
actually
the
original
code
name
for
this
project?
And
so
there's
been
a
lot
of
things
that
happen
between
the
original.
B
You
know
when
you're
working
in
an
organization
it's
important
to
figure
out
how
to
propagate
your
meme.
If
you
want
your
ideas
to
be
to
be
big-
and
I
think
marketing
is
an
important
part
of
that,
so
I
named
the
poll
request
project
tahoe
and
I
included
this
detail
that
I
I
had
learned
recently
and
I
you
know
I've
been
spending
a
bunch
of
time.
B
Tahoe
is
a
lake
in
california
for
those
of
you
who
aren't
like
super
up
on
californian
geometry
or
geography-
and
you
know,
there's
a
bunch
of
skiing
and
other
things,
so
I
had
actually
been
spending
a
bunch
of
time
up
there.
A
lot
of
the
the
spark
sql
catalyst
optimizer,
like
nested
columns,
were
implemented
in
a
cabin
in
tahoe,
so
it
has
a.
C
B
Is
an
absolutely
massive
lake
and
I
was
trying
to
build
an
absolutely
massive
data.
Lake
tahoe
is
so
large
that
it's
actually
not
that
large.
But
it's
incredibly
deep
and
it
has
so
much
water
in
it
that
if
you
were
to
spread
it
out
evenly
over
the
entire
state
of
california,
it
would
still
be
over
a
foot
deep,
which
I
think
is
like
a
pretty
crazy
amount
of
water.
And
I
just
thought
that
was
really
cool.
And
so
I
I
included
that
in
the
the
first
opening
pr
description.
A
That's
awesome,
so
there
we
go
now
you
have
the
context,
and
so
we
got
some
other
questions
coming
in,
but
I
did
want
to
add
one
for
td,
just
because
I
know
both
td
and
michael
are
going
to
be
going.
So
from
your
perspective,
since
you
know
td,
since
you've
been
doing
all
you
know
a
little
bit
about
streaming,
you
know
last
time
I
checked
so
I'm
just
curious.
A
H
As
michael
said,
that
the
apache
spots
file
stream
sync
solved
pieces
of
the
problem
of
writing
out
files
from
a
structure
streaming
query
into
a
into
some
sort
of
cloud
storage
or
distributed
storage,
but
it
had
major
limitations
in
terms
that
you
could
not.
You
could
not
write
from
multiple
streams
to
the
same
location
or
you
could
not
combine
streaming
writers
and
bad
job
writers
to
the
same
location,
same
table
using
anything
that
was
present
out
there.
H
So,
from
streaming
point
of
view,
this
expanded
on
the
kind
of
use
cases
you
could
use
you
can,
you
could
use
structure
streaming
with,
for
example,
be
it
multiple
streaming
queries
up
in
the
same
cable
or
streaming
query,
generating
key
value
data
you
want
to
absert
into
a
delta
table
into
a
location
that
delta
table
gave
you
the
ability
to
do
margins
and
upsells
into
it.
So
all
of
that
use
cases
that
was
not
possible
before,
with
a
more
managed
layout
format
like
delta,
it
just
became
possible
so
yeah
it
just.
A
A
We
have
still
plenty
of
experts
for
for
uw,
so
keep
on
asking
your
questions,
so
I'm
gonna
go
switch
gears
and
by
the
way,
for
the
people
who
are
already
asking
questions
on
linkedin
and
youtube
and
wherever
else
keep
on
asking
them,
we
will
answer
them.
So
all
right
take
care.
Michael,
take
care
td.
H
D
Yeah,
I
think,
there's
also
another
interesting
thing
around
like
streaming.
Workloads
specifically
with
delta
is
that
you
know
streaming
workloads.
Have
this
inherent
trade-off
between
latency
and
throughput?
Like
do
you
want
to
be
able
to
access
your
latest
data
as
soon
as
possible?
Or
do
you
want
to
be
able
to
query
your
table?
You
know
as
efficiently
and
as
effectively
as
possible.
So
this
you
know
inherent
latency
where
you
want
to
be
like
providing
the
access
to
the
latest
data.
You
need
to
write
data
as
soon
as
possible.
D
That
means
you
don't
get
to
write
out
large
files,
and
you
know
this
problem
with
like
cloud
object
stores.
Is
that
if
you
don't
write
out
large
files,
you
have
this
like
hit
in
terms
of
query
performance,
where
you
need
to
open
up
a
lot
of
connections
to
read
those
files
and
your
queries
like
over
time
get
slower
and
slower.
So
one
of
the
also
cool
things
about
delta
was
that
you
could
do
you
know
compactions
on
your
data
as
a
you
know,
different
writer
than
your
original,
like
streaming
writers
or
your.
D
A
Awesome
stuff,
thank
you
very
much
for
that
context.
Barack,
really
appreciate
it
man.
So
let
me
go
to
the
next
question
then,
actually
so
tyler,
I'm
gonna
target
this
question
to
you.
If
that's
okay
from
your
persp
perspective,
you
know
why
was
delta
lake
necessary
like
what
problems
did
it
solve
for
you
actually.
F
I
usually
didn't
know
about
delta
lake
until
mid
2019,
when
a
bunch
of
people
went
to
spark.
An
ai
submit
and
came
came
back
like
oh,
my
gosh
delta
lake.
Oh
my
gosh
delta,
like
let's,
we
need
to
use
delta
lake,
we're
actually
planning
a
move
from
an
on-premise
infrastructure
to
aws
in
s3,
and
probably
the
number
one
thing
that
delta
lake
solved
for
us
kind
of
out
of
the
box
was
this
question
of
consistency.
F
Aws
s3
has
gotten
a
lot
better
in
the
last
couple
of
years
becoming
more
consistent,
but
it
is
still
fundamentally
eventually
consistent
api
and
what
people
would
do
is
they
deploy
like
s3
guard
or
other
things
to
where
you
could
have
different
different
readers
and
writers.
Working
with
data
in
s3
and
delta
lake
just
disappeared.
All
those
problems
which
was
pretty
pretty
nice
we're
also
looking
at
streaming
use
cases
in
batch
use
cases,
and
so
we
were
planning
like.
F
Oh
all
right,
our
streaming
jobs,
they're
going
to
talk
to
kafka
and
then
they're
going
to
put
data
over
here
and
our
batch
jobs
are
going
to
go
mess
with
data
over
here
and
because
we
can
have
batch
jobs
and
streaming
jobs
both
reading
and
writing
from
delta
lake
tables
simultaneously
effectively.
It
also
unified
the
the
data
layer
quite
a
bit
to
where
everything
everything
sort
of
comes
into
delta
lake
and
goes
out
of
delta
lake.
There's
no
sort
of
separate
data
streams
in
the
data
platform
anymore.
A
All
right
then
relate
to
this,
which
I
know
tyler
you're
integrally
involved
with,
but
I
definitely
want
to
have
qp
kick
it
off,
first,
which
you
would
have
asked
me
to
do
anyways,
which
is
so.
Why
did
you
decide
to
then
support
the
delta
like
delta
rust,
api
right?
How
did
what
did
was
it
intended
to
solve
when
you
now
that
you're
using
delta
lake,
but
then
you've
wanted
to
basically
build
the
rest
api
to
go
with
it
so,
like
yeah,
why
did
you
do
it?
E
So
at
squid
we
do
have
a
lot
of
other
type
of
consumers,
adult
potential
data
like
consumers,
for
example,
we
have
a
lot
of
rubid
code
code
base.
We
have
a
python
codebase
and
a
lot
and
go
coolbase
as
well,
so
all
of
these
they
won't
be
able
to
access
delta
lake
because
we're
not
running
a
jvm,
we're
not
running
in
spark.
So
the
well.
E
So
this
also
for
my
for
myself,
it's
sort
of
it's
like
a
learning
project.
For
me,
this
seems
to
be
the
trend
for
a
lot
of
us
projects
where,
for
example,
the
official
aws
sdk
for
us
was
started
as
a
learning
project
for
the
original
maintainer.
Now
it's
one
of
the
official
projects,
so
for
me
it's
more.
It's
also
like
oh
well,
we're
using
delta
at
script.
Maybe
I
should
learn
more
about
this
thing,
so
I
spent
weekends
hacking
on
this
and
got
it
working.
E
F
Thank
you
very
much.
The
original
idea,
like
I
had
pitched
the
original
idea
in
april
of
last
year
and
then
qp
like
this.
Wasn't
I
didn't
ask
qp
like
hey?
Do
you
think
we
could
start
working
on
this
or
anything
like
that?
I
almost
feel
like
I
nerd
sniped
them
to
where
I
was
like.
Oh
there's
going
to
be
a
lot
of
engineering
work
to
do
this
delta
implementation,
then
qp
goes
and
reads
the
protocol
and
the
protocol
is
open.
It's
under
delta
io
in
the
delta
repository
the
protocol
is
not
that
complex.
F
It's
actually
like.
I
had
actually
never
spent
the
time
to
look
at
it
and
it's
definitely
worth
a
read
for
anybody
doing.
You
know
non-trivial
work
on
top
of
delta
like
just
to
understand
what
what
actually
happens
underneath
the
covers,
because,
like
delta
lake,
is
not
my
sequel,
it's
not
it's
not.
You
know
a
database,
but
the
behaviors
that
you
you
get
comfortable
with
on
delta
lake
can
be
informed
by
understanding
the
actual
protocol
and
what
goes
into
the
delta
transaction
log.
A
Perfect
thanks
very
much
tyler
all
right.
Let
me
actually
target
this
question
to
ryan,
because
ryan's
actually
been
quiet.
That's
not
a
good
thing
buddy.
So
I'm
just
curious,
like
you
know,
from
the
perspective
of
other
systems
connecting
the
delta
leg,
so
we
just
asked
qp
and
tyler
about,
like
you
know,
using
other
languages
like
rust
and
the
for
all
for
those
folks
that
don't
know
because
of
the
rust
binding,
the
rust
api
you.
It
also
has
python
bindings
and
soon
we'll
have
golang
and
ruby.
A
I
believe
right
so
yeah
so,
but
the
context
is
that
basically,
it
allows
other
your
programming
language
of
choice,
but
then
the
question
I'm
gonna
and
actually
ask
to
you-
which
of
course
is
a
tad
deleting
here.
You
know
how
about
other
systems
like
whether
it's
your
presto,
your
hive
or
your
athena's,
like
how
do
they
connect
the
delta
lake
right?
Because
it's
not
just
about
spark
anymore
as
much
as
you
know,
we
love
spark.
A
G
Yeah
so
currently
delta
actually
have
like
a
hive
connector,
you
can
use
half
read
delta
tables
and
in
addition,
we
also
build
like
a
delta
standalone
reader,
which
you
can
read
a
data
table
without
spark.
So
anyone
can
use
this
library
to
like
build
a
connector
for
other
systems.
So
for
presto.
Currently
we
have
a
spatial
integration
like
manifest
files,
which
is
kind
of
regenerator.
G
A
Soon,
excellent
thanks
very
much
okay,
so
we
have
all
those
whole
slow
questions
coming
in
from
linkedin
youtube,
and
so
let
me
go
ahead
and
start
trying
to
like
pull
some
of
those
questions
in
and
this
will
be
open
to
all
of
us
so
feel
comfortable
to
just
chime
in
whenever
you
want
okay.
So
one
of
the
first
questions
I'm
pulling
this
from
linkedin
is
there
a
benefit
of
creating
a
data
frame
from
a
delta
table
rather
than
the
table
created
from
parquet
files?
A
G
Yeah-
I
can
answer
this,
so
you
it's
basically
reading
a
data
frame
from
either
a
delta
or
a
package
format
like
this.
So
when
you
look
like
when
you
try
to
use
like
a
data
phone
api
to
load
a
packet
table
using
spark,
it's
probably
where
you
need
to
like
the
info
packet
file
schema
and
also
need
to
like
a
list
of
files
in
your
table
and
for
some
way
your
table
is
very
large.
This
will
become
very
slow
and
for
delta.
G
We
have
everything
in
the
transaction
lock,
we
can
just
use
spark
itself
middle
east
transaction
lock.
This
means
reading
transaction
knob
is
also
itself.
We
are
using
of
all
of
your
cluster
results
to
speed
up
so
and
then
we
can
quickly
figure
out
which
file
we
should
read,
and
then
we
can
just
return
all
the
metadata
queries
will
be
very
fast.
F
A
D
Yeah,
all
right
so
delta
supports
a
merge
operation
that
you
know
you
can
use
to
actually
perform
upserts
on
your
table.
So
you
know
this
is
like
a
very
standard,
sql,
syntax
to
actually
say
hey.
You
know
I
want
to
merge
my
source
data
into
my
target
table.
Here's
my
like
key
condition.
You
give
like
a
joy,
join
condition
on.
D
D
On
top
of
that,
you
know
the
way
it
works
is
essentially,
it
does
kind
of
like
a
copy
on
write
semantic.
Where
you
know
we
can
read
the
files
change
the
records
that
need
to
be
changed
or
updated
and
then
write
the
files
back
out
again
and
that's
kind
of
like
how
tahoe's
like
protocol
works.
Delta's
protocol
works
because
you
know
files
are
essentially
immutable.
You
can't
go
and
like
change
the
records
in
them
and
then
overwrite
them.
A
Awesome,
thank
you
very
much,
barack
okay,
one
more
from
linkedin
before
I
switch
to
the
other
other
forums,
and
I
believe
somebody
had
asked
this
because
it
wasn't
from
us,
even
though
it
says
it's
from
us,
but
can
I
read
delta
without
spark?
Can
I
use
a
single
node
just
to
read
it
is
delta,
coupled
with
spark
I'm
definitely
going
to
be
targeting
qp
or
tyler.
To
answer
that
question
I'll,
let
tyler
speak
first.
F
Oh,
you
stinker,
you
take
it
so
with
delta
rs
you
you
don't
need
spark
anymore
to
read:
to
read
delta
lake
there's
also
a
java
standalone
reader
as
well
for
reading
delta
lake
on
top
of
delta
rs.
There's
a
python
binding.
So
if
you
can
use
this
right
now,
if
you
do
a
pip,
install
delta
like
you'll,
be
able
to
use
from
your
single
local
machine
or
a
single
machine
in
a
cloud
provider.
F
Access
a
delta
table
and
the
python
binding
is
actually
really
cool.
There's
a
contributor
named
florian
who
did
a
lot
of
the
work
there,
the
that
python
binding,
allows
you
to
take
a
delta
table
and
just
convert
it
into
a
pandas
data
frame,
and
so,
if
you're,
already
comfortable
doing
some
amount
of
data
processing
in
python
natively,
you
can
just
take
a
delta
table
and
turn
it
into
a
pandas
data
frame
and
start
querying,
which
is
pretty
cool.
E
A
Got
it
thanks
very
much
qp,
okay
cool,
let's
switch
over.
We
have
another
one
question
from
youtube,
so
let
me
go
ahead
and
do
that
and
the
question
is:
is
delta
lake
now
supported
over
gcs?
Anybody
would
like
to
take
that
or
I
can
take
it
for
that
matter
as
well.
So
just
wanna
ask
that
question.
First,.
A
All
right,
so
the
quick
answer
to
that
question
is
then:
yes,
delta
lake
actually
works
over
google
cloud
storage.
It
actually
worked
even
before
we
introduced
data
bricks
on
google.
So
the
fact
is
that
you
could
always
already
hit
google
cloud
storage,
but
now
that
we
actually
are
running
data
bricks
on
google,
it's
even
better
still.
So
that's
the
quick
answer,
there's
not
much
more
to
add
because
it
just
works.
F
A
F
F
A
A
Basically
these
transactional
systems,
like,
in
other
words,
database
type
systems
into
delta
lake,
with
low
latency
like
30
to
60
seconds.
So
that's
the
definition
of
low
latency
in
this
case,
so
different
people
have
different
latent
definitions
low
length.
So
this
is
their
definition,
at
least
with
minimal
data
engineering,
in
other
words,
what's
the
the
least
amount
of
code
that
you
can
go
ahead
and
to
basically
allow
that
type
of
movement
to
you
know
to
move
that
data
from
your
transactional
system
into
a
delta
lake.
F
Yeah
so
there's
there's
work,
that's
scripted,
mostly
by
one
of
the
folks
on
our
data
engineering
team,
the
sql
delta
import,
which,
which
ryan's
passing
me
that
question
because
he's
busy
hitting
the
merge
button,
I'm
sure
on
that
poor
request,
but
sql
delta
import.
We
we
develop
because
we're
doing
nightly
imports
from
relational
data
stores
into
delta
lake,
but
because
the
question
is
really
around,
I
would
say
something
approaching
a
little
bit
more
of
a
real
time
I'll.
We
haven't
done
that
yet.
F
But
I'll
tell
you
what
we
were
thinking
of
doing
so
key
in
our
architecture
is
kafka
for
a
lot
of
our
streaming
workloads,
and
we
already
have
some
tooling
that's
present
to
take.
You
know,
messages,
json,
formatted
messages
coming
in
on
kafka
and
insert
each
message
as
a
row
into
a
configured
delta
table.
This
was
the
original
inspiration
for
the
kafka
delta
ingest
project,
which
is
now
open
source
in
the
delta
lake
project.
So
we
already
have
the
the
right
side.
I
would
say
of
that
equation
where
you
know.
F
F
Was
deploying
a
tool
called
dibyzium,
which
is
d-e-b-e-z-I-u-m,
which
you
can
connect
to
a
relational
data
store
to
start
streaming,
changes
off
of
that
data
store.
The
reason
we
haven't
done
that
yet
is
our
relational
data
models
are
not
so
great,
there's
not
enough
data
engineering.
That's
actually
gone
into
those.
So
when
we
were
looking
at
streaming
data
out
of
those
relational
databases,
we're
gonna
have
to
do
a
tremendous
amount
of
transformation
of
that
data
anyways.
So
we
we
actually
shied
away
from
it.
F
But
if
you
were
to
go
down
that
route,
I
would
suggest
looking
at
division.
You
know
division
kafka
and
then
something
to
take
data
from
kafka
and
bring
that
into
delta
lake.
You
can,
I
mean
a
simple
spark
streaming
job.
Can
you
know
stream
from
kafka
and
and
stream
it
into
delta
lake,
in
like
25
30
lines
of
spark,
but
that's
the
architecture?
I
would
look
at
right
now.
A
A
Oh,
that's
right,
good,
call
out
brock
cool
all
right,
let's
catch
up,
because
there's
still
a
ton
of
questions
and
let
me
go
back
to
linkedin
before
we
lose
those
folks.
So
we
got
a
great
question
here:
does
delta
lake
have
versioning
and
in
this
case
I'm
going
to
presume
data
version?
So
let's
go
with
that.
D
Yeah,
I
can
jump
on
this
question,
so
the
in
delta
tables,
like
every
single
commit,
is
essentially
a
version
of
the
table.
So
what
happens?
Is
you
know
every
you
know,
change
that
you
do.
It
appears
as
kind
of
like
a
delta
in
the
transaction
log,
so
that's
also
somewhat
related
to
where
the
name
comes
from,
but
it
really
isn't
like
where
the
delta
name
actually
came
from.
But
you
know
these
deltas
essentially
make
up.
D
You
know
each
version
of
your
table,
so
what
you
can
do
after
that
is
you
know
you
can
compute
the
state
of
the
table
at
any
given
version
and
when
you
can
do
that,
you
can
also
just
essentially
go
back
in
time.
Look
at
how
your
table
was
a
day
ago
a
month
ago.
Maybe
even
you
know
a
year
ago
or
you
can,
you
know
just
always
query
the
latest
version,
so
it's
kind
of
like
a
technique
called,
as
I
mentioned,
like
mvcc,
and
you
know
this
allows
kind
of.
D
Like
you
know,
as
new
data
comes
in,
you
know
the
readers
that
actually
queried
the
table.
They
don't
have
to
see.
You
know
all
the
changes
that
are
being
made
to
the
table
in
real
time.
What
they
can
do
is
they
can
just
like
you
know,
query
the
table
at
the
version
that
they
were.
You
know
they
started
the
query
so.
D
Okay
yeah,
so
the
delta
name
was
actually
quite
interesting.
I
mean
it
happened
like
around
kind
of
like
new
year's
time,
and
you
know
I
came
back
to
work
and
you
know
tahoe
suddenly
became
delta.
I
was
like,
where
did
delta
come
from?
You
know
I
was
like
initially
we
were
thinking
like
reservoir,
but
you
know
reservoir
was
both
very
hard
to
write
and
also
hard
to
pronounce.
D
D
You
know,
you
know
like
more
millennial
style,
but
you
know
in
the
end,
like
one
of
our
you
know,
members
at
databricks
jules
he
actually
came
up
with
this
name
delta,
because
you
know
the
idea
is
you
know
you
have
these
deltas
that
you
know
rivers
flew
into
and
that's
where
all
the
sediments
you
know.
Eventually,
you
know
build
up
and
that's
where
you
you
know,
and
then
you
know
once
you,
the
stream
goes
into
a
delta.
D
It
splits
off
into
like
different
streams,
and
you
know
you
have
this
massive
sediment
buildup
and
that's
where
all
your
crops
grow.
That's
where,
like
most
quality
you
get
from
that
river.
You
know
right
at
this
like
offset
in
this
delta,
so
we
thought
you
know
that
was
actually
a
very
nice
very
nice.
You
know
like
metaphor
to
think
about.
D
You,
know
kind
of
like
streams
coming
in
and
you
know
like
sediments
and
quality,
and
you
know
it's
kind
of
like
controlled
and
it
also
fit
into
our
architecture
of
you
know
like
you,
can
have
massive
firepower
streams
going
in
and
then
suddenly
splitting
out
into.
You
know
like
materialized
views
and
things
like
that,
so
it
felt
quite
natural.
A
Awesome,
no.
Thank
you
very
much.
I
always
love
that
story.
Dude
all
right.
Next,
on
the
line
for
questions,
there
is
I'll
just
answer
this
one
real
quick,
but
is
there?
How
is
data
bricks?
No,
that's
not
the
one
I
was
looking
for
sorry.
You
know
what
no
let
me
switch
to
a
different
question.
It's
also
on
here
on
linkedin.
How
does
delta
actually
handle
upserts
we,
so
we
already
talked
a
little
bit
about
the
merge.
So
let
me
so.
A
I
think
we
already
technically
answered
that
part
of
the
question
but
like
how
far
can
you
go
back
like
you
know?
Can
you,
for
example,
can
you
go
back
a
couple
of
years
of
the
for
of
your
data?
If
you
wanted
to
just
since
you've
brock,
you
already
called
out
the
mbcc
portion,
one
provided
a
little
guidance
on
how
far
back
in
time
you
know
based
on
size
or
maybe
should
we
use
some
other
alternatives,
for
example,
to
compensate
for
that.
A
D
D
So
you
know
there's
this
kind
of
like
trade-off
where,
if
your
table
is
changing,
you
know
once
every
you
know
minute,
you
might
have
a
ton
of
json
files
to
read
if
you
want
to
go
back
in
time
for
essentially
a
year,
because
you
know
for
also
cost
reasons,
we
clean
up
some
of
the
checkpoint
files,
because
these
checkpoint
files
can
actually
add
up
to
you
know
quite
a
lot.
You
know
quite
large
sizes.
D
So
you
know
if
your
data
is
changing
once
a
day,
then
it's
not
actually
that
bad
to
go
back.
You
know
three
five
years,
but
if
you
want
to
go
back,
you
know
five
years,
but
your
data's
changing
every
minute
or
every
five
minutes
or
every
hour
it's
actually
better
to
have
kind
of
like
an
archive
table
that
can
you
know,
take
your
daily
snapshots
from
delta
and
use
that
to
actually
go
back
those
three
five
years.
You
know
if
you
need
that
for
like
regulatory
reasons
or
whatnot.
F
F
What
that
means,
though,
is
that
the
delta
log
has
to
have
record
of
whatever
data
that
you're
trying
to
look
at,
and
so,
if
you
think
about
you,
know
our
use
case
where
we're
streaming
data
into
a
table,
we're
adding
files
we're
adding
files
we're
adding
files,
we
might
go
in
and
delete
files
as
well.
The
transaction
log
is
going
to
have,
you
know,
add
add,
add,
remove,
add,
add,
and
if
I
look
at
that,
I
can
travel
back
to
before
or
after
the
remove.
F
Unless
I
execute
a
vacuum
because
the
vacuum
is
going
to
come
in
and
it's
going
to
delete
files,
you
know
it's
going
to.
Basically
you
know
anything
that
I've
removed,
that
I
don't
need
anymore,
it's
going
to
get
rid
of,
which
is
really
good
for
that
sort
of
cost
trade-off.
That
barack
was
mentioning.
So
if
you,
if
you
find
yourself
executing
vacuums
with
some
regularity
and
in
scripps
use
cases,
we
do
this
on
some
tables.
F
F
We
use
there's
a
really
useful
pattern
that
we
use
for
streaming
tables,
where
we
will
stream
data
into
a
delta
table
and
then
every
night,
because
we're
we're
basically
partitioning
on
a
date
stamp.
So
every
single
day,
we'll
have
one
one
partition
of
data
that
night
after
a
full
day
of
streaming
has
been
completed.
F
I
think
for
how
how
efficient
or
how
long
you
want
to
support
time
travel.
We've
had
some
users
request
like
hey.
I
want
to
go
back
to
the
beginning
of
time.
No,
no,
we're
not
going
to
support
that.
We
can
give
you
certain
time
slices,
because
you
know:
there's:
there's
cost
trade-offs.
There's
performance
trade-off
that
we've
got
to
make
alongside
that.
E
On
the
the
clone
operation,
there
are
actually
two
types
of
clones.
One
is
deep
and
one
is
shallow
because
we
actually
keep
track
of
the
files
using
the
transaction
log.
If
you
do
a
shallow
clone,
it's
actually
really
cheap.
It's
really
just
cloning,
the
pointers.
We
don't
actually
copy
the
data,
but
if
you
do
a
deep
clone,
that's
where
you
keep
get
copied
all
the
data
as
well,
and
so
a
dip
clone
will
not
be
impacted
by
vacuum.
E
A
Excellent
we've
got
some
pretty
awesome
stuff
and
a
good
shout
out
to
cloning
as
well.
So
all
right.
Well,
then,
actually,
I
think
this
actually
segues
to
another
question.
I
think
it
came
in
I'm
actually
forgetting
where
I
came
from
now.
I
apologize,
but
this
is
when
you're
doing
machine
learning
actually
does
it
make
sense
to
use
delta
lake
anybody
like
to
chime
in
on
that
one.
D
Oh,
I
can
talk
about
kind
of
like
the
integrations
that
delta
has
with
ml
flow,
so
ml
flow.
Has
this
auto
logging
feature
where
you
know
it
keeps
track
of
like
which
models
you're
training?
You
know
which
algorithm
you're,
using
which
hyper
parameters
you're
using
to
tune?
You
know
your
you
know
model
with,
and
you
know
as
part
of
this
autolog
auto
logging
feature.
You
know
when
you're
querying
a
delta
table,
it'll
also
keep
track
of
which
version
your
delta
table,
which
version
of
your
delta
table
was
used.
D
So
you
know,
ml
flow
can
track
all
these.
You
know
experiment
data,
you
know
in
its
server,
so
you
know
you
don't
have
actually
have
to
have
a
csv
file
or
excel
sheet
on
your
local
laptop,
which
you
know
only
you
can
see
and
no
one
else
can
see
you
get
all
this
information
and
you
can
also
reproduce
these
experiments
with
the
exact
same
input
parameters,
and
so
this
really
helps
you
know,
customers,
you
know
both
develop
their
machine
learning
pipelines.
D
D
D
So
you
want
to
see
next
year
if
you
can
perform
better,
so
what
people
do
is
they
can
actually
archive
that
version
of
the
table
using
a
deep
clone
as
well
and
and
then
you
know
next
year,
when
they
add
more
features
when
they
come
up
with
like
new
techniques,
they
can
go
and
hit
that
archive
table
instead
and
see
how
their
model
performs
you
know
happens
to
apples.
Instead
of
you
know,
this
year's
data
may
be
a
lot
different
than
you
know.
Last
year's
data
so.
E
A
Excellent
excellent
and
so
hey
a
couple
comments
or
call-outs.
Actually
I'd
like
to
add
to
this,
so
one
was
our
friend
andre.
He
wanted
to
call
out
the
fact
that,
with
respect
to
machine
learning,
delta,
no
flow
actually
allows
you
to
reproduce
an
experiment
with
the
exact
same
data,
which
is
exactly
the
call
out
here
right,
because
one
of
the
things
that's
sort
of
being
called
out
here
by
about
everybody,
is
this
context
of
data
drift
or
model
drift
right?
A
That
data
changes
over
time
and
with
that
your
model
itself
might
change
over
time
with
it.
If
you're,
not
aware
of
that
fact,
what
ends
up
happening
is
that
your
model
is
not
going
to
be,
as
will
work
as
well,
because
well,
the
underlying
data
had
changed
and
you
haven't
retrained
or
modified
or
updated
the
hyper
parameters,
according
right
so
having
the
original
data,
so
you
can
reproduce
the
experiment,
and
then
you
can
go
ahead
and
figure
out
how
to
adjust
accordingly
to
well.
You
know
get
a
do
a
better
job
with
it.
A
The
context
also
is-
and
I
think,
michael's
the
one
who
actually
mentioned
this
in
a
couple-
a
couple
of
blog
posts
or
videos-
is
that
underneath
the
covers
of
all
this-
is
that,
in
order
to
have
good
machine
learning,
you
have
to
have
good
data
quality
right
and
the
reality
is
with
delta
lake
you're.
Actually
able
to
do
that.
A
You
have
a
transaction
that
protects
the
data
as
it's
coming
in
you
you're,
preventing
corruptions
you
ensuring
the
schema
of
your
data
is
valid
right,
so
you're
not
going
to
have
the
wrong
columns
with
the
wrong
data
types
coming
in.
You
you're
able
to
ensure
that
the
quality
of
your
data
is
extremely
high.
Having
that
allows
you
to
have
better
machine
learning
models
right
and
so
a
lot
of
times.
A
You
don't
think
about
that,
and
it's
understandable
by
the
way,
because
most
non-production
environments
are,
I
download
a
file
that
has
every
all
the
data
clean
and
it's
easy
to
go
ahead
and
run
your
your
models
in
in
production
in
real
world
scenarios,
your
data
is
dirty
like
and
so
ensuring
that
quality
using
delta
lake
actually
allows
you
to
have
high
quality
data,
which
means
you
have
a
better
model.
So
anyways.
That's
me
getting
off
my
soapbox
on
that
one.
So.
E
And
just
adding
to
what
barack
mentioned
earlier
in
the
traditional
world
you
have
to
make
the
trade-off
between.
Do
I
want
to
go
back
to
the
one
go
stream
right
with
delta
lake
tables,
you?
Actually,
you
can
start
with
batch
ml
pipelines
and
eventually
move
to
a
stream
streaming
ml
pipeline
without
changing
the
underlying
data
store
and
to
get
a
better.
You
know
latency
on
how
you
want
to
compute
all
the
other
features
or
inference.
So
that's
also
a
really
good
plus
for
using
data
tables.
A
Rock
on
rock
on
no
thank
you
very
much,
qp,
okay.
So
a
couple
more
questions
before
we
call
today,
but
we
have
a
great
question
from
linkedin
concerning
how
does
delta
lake
actually
handle
schema
enforcement
and
schema
evolution?
And
then
a
subsequent
question
to
this
also,
is
that
like
is
there
like
a
maximum
number
of
columns
involved
as
well
and
so
to
anybody
here
in
the
panel.
G
Yeah,
I
can
answer
this
so
basically
for
basic
schema
parts.
Basically,
every
time
when
you
write,
when
you
write
it
to
a
delta
table,
we
will
make
sure
the
data
you
write.
That's
compatible
with
the
schema
we
store
in
a
transaction
law,
so
we
always
stole
a
table
schema
in
a
transaction
log
as
the
truth,
so
we
don't
use
the
schema
in
parque
actually,
so
this
means
we
can
also
support.
G
A
D
On
top
of
that,
delta
also
supports
check
constraints
so,
for
example,
to
keep
kind
of
like
your
data
quality,
you
know
as
high
as
possible.
You
can
say
that
you
know
hey.
My
timestamps
cannot
be
more
than
five
years
ago
because
hey,
I
didn't
launch
any
software
more
than
five
years
ago,
or
you
know.
Basically,
you
can
define
these
like
sql
expressions
on
your
table
that
you
know
make
sure
that
you
know
your
data
is
of
the
right
format
of
you
know
it's
been
parsed
correctly.
A
Records
excellent
thanks
very
much
brock,
all
right
cool
next
question
here
there
is
a
it's
more
of
a
generic
question,
but
I
think
it's
still
quite
valid,
so
I
figured
I'd,
ask
it!
This
is
from
the
zoom
chat.
It's
like.
Have
you
like,
worked
in
production?
Have
you
ever
implemented
a
streaming
pipeline
with
delta
that
processes
hundreds
of
thousands
of
transactions
per
second,
it
seems
like
it's
a
just
basically
trying
to
figure
out.
A
F
Cluster.
I'm
very
curious
to
hear
from
barack,
like
I'm
sure,
I'm
sure,
from
databricks
stream
team,
you've
seen
a
lot
bigger
customers,
but
we've
had
like
pushing
pushing
tens
of
thousands
of
requests
per
second
through
spark.
Streaming
has
been
a
learning
experience,
I
think,
for
everyone
involved
here
on
on
figuring
out
how
to
tune
and
and
work
with,
spark
streaming.
D
D
Actually,
so
it's
a
matter
of
you
know
the
latencies,
though,
to
get
you
know
that
amount
of
throughput
you
might
want
to
think
about
in
the
latency
of
order
of
you
know
15
seconds
or
you
know
almost
a
minute
I
would
say
so.
If
you're,
okay,
with
kind
of
like
a
minute,
you
know
latency,
you
can
kind
of
hit
those.
D
You
know
a
million
records
or
you
know
three
million
records
per
second,
if
you
can
also
scale
out
kind
of
like
your
executors
really
well-
and
you
know
we-
you
know
these
pipelines
have
typically
been.
You
know,
pipelines
where
files
land
on
cloud
object,
storage
systems-
and
you
know
it
has
been
very
well
tuned
to
actually
maximize
the
throughput
from
these.
You
know,
file
object
stores,
so
it's
not
possible.
It
does
take
a
lot
of
work.
Maintaining
you
know,
and
you
know,
if
you're
fine,
with
kind
of
like
a
minute
latency,
that's
possible.
D
I've
also
seen
some
like
kafka
pipelines
where
we
were
able
to
get.
You
know
like
1.2
1.3,
millions
of
records
per
second
with
structured
streaming
as
well
as
delta.
But
again
you
know
you
need
to
realize
that
you're
essentially
writing
to
files.
Then
you're
reading
back
from
the
files
you're,
you
know
paying
this
cost
of
like
serialization.
You
know
part
k,
serialization,
part
d,
serialization,
and
then
you
know
is
that
good
enough
to
you
know,
send
an
alert
around
like
fraud.
You
know
if
it's
okay
to
send
a
customer.
D
You
know
a
minute
later:
hey
we
detected
this
transaction
from
your
credit
card.
Is
this
okay?
Then
you
know
tell
us
no,
and
then
you
know
we'll
take
action
or
something.
If
that's
fine
yeah,
you
can
definitely
go
ahead.
If
that
you
know,
if
you're
looking
for
something
like
no,
I
need
to
do
this.
You
know
like
within
the
second.
You
know:
kafka
to
kafka
run
something
well.
You
know,
then
delta
doesn't
come
into
play
here.
It's
like
you,
know,
you're,
just
going
from
cough
with
kaka.
At
that
point,
I'd.
F
Add
that
if
you're
doing,
if
you're
doing
high
latent
or
low
latency,
you
know
high
throughput
streams,
I
would
definitely
look
at
how
you
can
instrument
the
spark
streaming
jobs
for
your
own
analytics
needs
like
you're
not
going
to
get
the
stuff
that
you
might.
You
might
want
out
of
spark
ui
at
that
scale
or
that
that
high
latency,
so
what
we've
done?
Qp
actually
wrote
a
blog
post
about
this.
We
integrate
datadog
with
our
spark
streaming
jobs
so
that
we
can
pull
some
of
those
metrics
out
into
datadogs.
A
Perfect,
thank
you
very
much
guys.
Okay,
I
want
to
cut
short
just
because
we
only
have
a
minute
left
before
we
are.
We
are
done
so
for
starters,
I
want
to
say
thank
you
very
much
for
everybody
for
for
me,
for
speaking
at
today's
panel,
really
appreciate
your
time
to
all
you
folks.
I
apologize
for
not
getting
to
all
your
questions.
We
had
a
lot
of
questions
coming
in
from
all
three
different
venues,
so
it's
hard
to
keep
track
of
all
that.
A
So
my
apologies,
but
if
you
do
have
any
more
questions,
don't
forget
to
join
us
on
delta
dot,
io,
that's
the
delta
lake
website
at
the
bottom
or
just
click
on
community.
You
can
find
us
on
the
google
groups.
You
can
find
the
databricks
youtube
channel.
You
can
also
find
us
on
our
slack,
so
you
can
talk
to
us
through
that
way.
So
again,
so
please
go
ahead
and
join
the
community.
A
Continue
the
conversation,
as
you
can
tell
a
lot
of
us
love
answering
questions
so
by
all
means,
go
ahead
and
talk
to
us
there
so
again,
apologies
for
not
tackling
all
your
questions,
but
you
can
find
us
there
and
you
can
berate.
You
can
certainly
break
me
at
least
there,
okay
and
then
one
more
call
out
for
the
data
and
ai
summit.
Don't
forget
that
we're
all
gonna
be
there
so
again,
so
you
can
chime
us
there.
A
You
can
look
us
up,
but
you
can
talk
to
us
that
way,
obviously
virtually
and
then
don't
forget
this
thursday
there's
also
a
bunch
of
really
cool
community
lightning
talks
so
be
on
the
lookout
for
that
and
join
us
there
all
right.
So
I
think
that's
it
for
today
again.
Thank
you
very
much
everybody.
I
appreciate
your
time
and
you
all
have
a
good
one.
All
right.