►
Description
We're starting to get feedback from the Delta Lake community on more integrations per the proposed 2021 H2 roadmap. Through this talk, Vini and Denny will provide a recap of the features including Spark 3.1 support, Delta Sharing released in 2021 H1, and what the community asks for the future roadmap for Delta Lake OSS. There are callouts for OPTIMIZE, Apache Heron, and Trino CTAS support as well as the current integration efforts around Apache Flink, PrestoDB, Apache Pulsar, LakeFS, and Nessie. Don't forget the standalone readers and writers and rust lang API optimizations
What are your favorites? Come participate, engage with the community and get your voices heard for Delta Lake!
Learn more: https://delta.io/
Join the Delta Lake community slack: https://dbricks.co/delta-users-slack
Join the Delta Lake google group: https://groups.google.com/g/delta-users
Roadmap Survey: https://forms.gle/LGMQtEbEjGezvfPz6
Meet our speakers:
Vini Jaiswal is a Developer Advocate at Databricks who helps data practitioners implement scalable data architectures and AI applications. She brings several years of Data and cloud experience working with Unicorns, Digital Natives, and some of the Fortune 500 companies including her role as VP - Data Science Engineering Lead at Citibank, Data Analyst at Southwest Airlines and holds an MS in Information Technology and Management from the University of Texas at Dallas.
Denny Lee is a Developer Advocate at Databricks. He is a hands-on distributed systems and data sciences engineer with extensive experience developing internet-scale infrastructure, data platforms, and predictive analytics systems for both on-premise and cloud environments. He also has a Masters of Biomedical Informatics from Oregon Health and Sciences University and has architected and implemented powerful data solutions for enterprise Healthcare customers. Databricks is proud to announce that Gartner has named us a Leader in both the 2021 Magic Quadrant for Cloud Database Management Systems and the 2021 Magic Quadrant for Data Science and Machine Learning Platforms. Download the reports here. https://databricks.com/databricks-named-leader-by-gartner
A
Awesome
and
then
so
right
before
I
pass
it
over
to
our
speakers.
I
just
wanted
to
mention
that
this
session
is
recorded
and
it'll
be
available
on
youtube
directly
after
the
broadcast
so
I'll
make
sure
after
I
stop
sharing
your
screen
I'll
share
all
the
links
to
the
programs
that
I
shared
and
then
feel
free
to
ask
questions
so
and
we're
also
moderating
the
chat
spaces.
So
ask
your
questions
there
and
I
think
that's
it
so
without
further
ado
vinnie,
please
take
it
away.
B
That's
exciting
hope
it
makes
people
who
are
joining
excited
to
you
know
just
collaborate
with
us,
so
let
me
share
my
screen
all
right
awesome.
So
if
you
can
see
my
screen,
we
are
here
in
today's
tech
talk.
We
will
provide
an
overview
of
delta
lake
oss.
My
name
is
vinnie
jaswal
and
I
am
a
developer
advocate
at
databricks.
B
C
B
Awesome
thanks
for
introduction
day,
so
we
will
start
with
what
the
delta
lake
is
for
those
who
are
new
to
it:
the
pace
of
innovation
in
delta
ecosystem
and
what
is
coming
up
in
the
road
map.
In
the
next
two
quarters,
we
will
wrap
up
the
session
with
few
more
call
outs
and
ways.
You
can
be
a
part
of
this
exciting
journey,
so
please
stay
tuned.
B
B
Some
challenges
where
delta
lake
is
used
is
around
malinform
data
ingestion
difficulties
deleting
data
from
for
compliance
or
issues
modifying
data
for
your
data
capture.
So
your
data
can
be
from
a
variety
of
different
sources
and
it
is
either
getting
ingested
as
a
batch
or
stream
in
the
delta
lake.
In
the
diagram
I
am
presenting
the
delta
leaf,
medallion
architecture,
so
in
dull
medallion
architecture
there
are
three
tables
categorized
as
bronze
silver
and
gold,
and
the
way
to
think
about
these
categories
are
first
is
injection
table.
B
B
So,
today,
delta
lake
is
used
all
over
the
world.
Since
you
are
all
joining
from
different
parts
of
the
world,
it
can
get
you
exciting
over
an
exabyte
of
data
per
day
gets
processed
on
delta
lake
and
that
accounts
for
75
percent
of
delta.
That
is,
delta
data,
that
is
scanned
on
databricks
platform
alone
and
delta
lake
has
been
deployed
to
at
least
more
than
3
000
customers
in
their
production
lake
house
architecture.
B
B
So
in
a
nutshell,
delta
lake
features
are:
it
provides
asset
transactions,
scalable
metadata
handling
data
versioning
in
form
of
snapshots,
which
provides
ability
to
time
travel
data
is
stored
in
open
format
and
delta
lake
unifies
streaming
and
batch
processing.
Further
delta
automatically
handles
schema
variations
to
prevent
insertion
of
bad
records
during
ingestion.
B
It
allows
you
to
audit
history
through
transaction
logs
delta
lake
also
supports
dml
commands,
including
delete
update
and
merge,
and
delta
lake
is
fully
compatible
with
apache
smart
api.
So
you
can
use
the
already
known
apache
spark
api
to
query
data
link,
and
these
are
some
of
the
examples
where
delta
lake
is
used
for
building
lake
houses.
B
Comcast
saw
massive
performance
gains,
replacing
640
machines
to
64
and
reduced
model
deployment
from
weeks
to
five
minutes,
so
delta
allowed
them
to
do
machine
learning
to
terabytes
of
data
from
over
20
million
devices
being
personalized,
bringing
personalized
content
to
viewers
based
on
their
voice
commands
salesforce
uses
delta
lake
for
their
engagement
activity
platform,
and
this
is
where,
like
customers
engage
and
their
engagement
mutation
job
can
now
handle
25
million
update,
delete
requests
within
15
minutes
which
outperform
their
goal
of
handling
10
million
mutations
within
an
hour.
That's
awesome
and
script.
B
They
simplified
the
management
of
their
data,
analytics
workflows,
significantly
improving
the
development,
velocity
and
cross-team
collaboration.
The
smile,
the
small
file
problem
that
played
their
engineering
team,
was
a
thing
in
the
past.
With
delta
lake,
they
enabled
performance
optimized
data
pipelines.
They
saw
huge
reduction
in
operational
and
infrastructure
cost,
which
equates
to
around
30
to
50
percent
cost
savings.
B
That's
exciting,
isn't
it
so,
let's
take
a
look
at
the
pace
of
innovation
in
delta
lake.
Today
we
are
at
delta
lake
1.0
version.
It's
been
a
long
journey
to
get
here.
Let's
take
a
look
back
at
where
we
have
began
and
figure
out
how
we
all
got
here.
So
delta
lake
was
first
open
source
in
april
of
2019
at
the
theta
and
ai
summit,
and
that
was
the
first
version
that
was
pretty
exciting.
It
included
asset
transactions
and
schema
management,
which
was
a
huge
improvement
in
the
world
of
big
data.
B
It
included
streaming
and
batch,
but
we
didn't
stop
there.
We
quickly
added
support
for
dml
commands
and
vacuuming.
We
added
scala
and
python
apis,
and
then
we
improved
compaction
and
concurrency.
It
was
made
possible
to
convert
parquet
files
into
delta
using
sql.
Only
other
things
that
were
added
in
the
next
version
are
things
like
describe
history,
which
allows
you
to
understand
how
your
table
has
evolved
over
the
past.
This
is
why
you
can
do
time
travel
in
0.7
version.
A
B
And
other
features
and
0.8
release
and
it
hasn't
stopped
us
from
there.
So
the
1.0
release
was
certified
by
the
community
four
months
ago
and
it
was
announced
at
data
and
ai
summit,
and
you
can
see
from
this
slide
that
there
are
a
variety
of
new
features
that
make
it
available
delta
available
everywhere.
So
let's
go
through
each
of
these
features
that
got
added.
B
The
first
feature
I
want
to
talk
about
is
generated
columns.
So
what
is
the
problem
we
are
trying
to
solve?
The
problem
that
generated
column
is
designed,
for
is
the
one
that
data
engineers
are
very
familiar
with,
so
you
have
a
table
that
has
a
timestamp
column,
but
you
don't
want
to
partition
by
timestamp
column,
because
that
would
result
in
way
too
many
partitions.
B
So
instead
you
want
to
partition
by
date,
but
to
do
that,
you
have
to
take
a
timestamp
convert
it
manually
into
a
date,
and
you
also
have
to
tell
your
users
that
they
need
to
manually
add
predicates
to
query
both
of
these.
This
has
some
query:
implica
computation
implications
as
well,
so
you
can
now
use
generated
columns
to
solve
all
this,
and
it
is
pretty
simple.
You
just
need
to
do
a
addition
of
little
bit
extra
information.
B
So
in
the
code
you
can
see
that
I
added
a
line
called
generated,
always
us
and
then
cast
event
time
given
time
as
date.
So
delta
understands
the
functional
dependencies
between
these
two
columns
and
it
would
enable
this
date
column
to
be
automatically
added
pretty
awesome.
Another
really
exciting
feature
in
delta.
1.0
is
cloud
independence.
B
Now.
What
do
I
mean
by
that
is
delta
has
always
worked
on
a
variety
of
storage
systems.
You
could
use
it
on
hadoop.
You
could
use
it
on
s3.
You
could
use
it
on
adls
gen
2,
but
with
the
advent
of
delta
1.0
and
the
delegating
log
store,
you
can
now
have
a
single
cluster
that
reads
and
writes
from
different
storage
systems.
This
is
pretty
awesome
because
it
means
that
you
could
do
federated
querying
across
your
data
stored
in
different
clouds,
or
you
can
use
this
for
cross
region
consolidation,
taking
information
from
all
different
places.
B
You
think
you
have
it
and
bringing
it
all
in
one
place,
and
this,
I
think,
is
pretty
cool
feature,
because
we
have
also
been
expanding
the
set
of
files
that
is
supported
by
delta.
So
we
added
support
for
google
cloud
and
also
ibm
cloud,
and
if
you
don't
see
your
favorite
one
in
here,
stay
tuned
towards
the
end
where
we
will
tell
you
how
you
can
get
involved.
B
Another
feature
is
multi-cluster:
transaction
delta
lake
always
had
support
for
multiple
clusters.
Writing
into
a
single
table
mediating
the
updates
with
asset
transaction
protocol
preventing
conflicts.
This
has
worked
on
hadoop,
hdfs,
adls,
gen,
2
and
now
google
cloud
storage.
However,
users
of
s3
have
always
been
left
in
the
dark.
B
It
turns
out
that
aws
s3
is
missing
the
transactional
primitives
that
is
needed
to
build
this
functionality
without
depending
on
external
systems,
and
this
is
why
I'm
excited
to
talk
about
a
couple
of
initiatives
from
our
open
source
contributors,
the
companies
scribd
and
samba
tv.
They
are
adding
support
in
the
delta
transaction
protocol
to
use
amazon
dynamodb
to
mediate
between
multiple
riders
of
amazon,
s3
endpoints.
Now,
multiple
delta
late
clusters
can
read
and
write
from
the
same
table.
B
That's
pretty
cool
now
that
we
have
got
great
python
support
as
well
in
delta.
Previously
we
mentioned
few
other
things
that
were
coming
up
in
the
roadmap,
but
yeah
with
now
with
titan
support.
Now
it's
important
that
it's
easier
for
python
users
to
get
started.
There
are
two
different
packages
depending
on
how
you
are
going
to
be
using
delta
lake
from
python.
B
If
you
want
to
use
it
along
with
spark,
you
can
just
do
pip
install
delta
lake
spark
and
it
will
set
up
everything
you
need
to
run
job
against
your
telelate.
But
if
you're
going
to
be
working
with
smaller
data
or
if
you
want
to
use
pandas
or
some
other
libraries
you
need
to
you
need
you
don't
need
spark
to
access
delta
from
python
anymore.
B
B
B
Another
place
where
we
did.
Some
pretty
deep
integrations
is
with
sql
compliance,
so
it
has
now
reached
ansi
sql
compliance
for
ddl
and
ml
dml
commands
like
insert,
merge
and
explain,
and
finally,
we
are
also
integrating
with
the
catalog
apis
and
delta
lake
has
integrated
with
those
to
ensure
data
tables
are
available
for
streaming
that
are
available
in
the
catalog
so
that
you
don't
have
to
handle
path
metadata
manually.
B
B
C
B
B
That's
pretty
awesome
right,
so
I
will
talk
about
what
it
does
with
communities,
interest
and
excitement
to
expand
delta
lake
and
make
it
available
everywhere
and
with
the
innovation
happening
in
the
delta
standalone
reader
writer.
This
is
how
we
are
expanding
a
community.
We
are
adding
enhancements
and
whole
bunch
of
connectors
to
delta
lake,
so
let
me
go
through
each
of
them
and
again
like
if
you
don't
see
your
favorite
one
here,
please
engage
with
us
on
what
you
want
to
get
at
it.
B
So
the
first
one
I
want
to
talk
about
is
apache
flink.
For
those
who
don't
know,
I
will
just
explain
about
each
connector,
so
apache
flink
is
a
framework
and
distributed
processing
engine
for
stateful
computations
over
unbounded
and
bounded
data
streams.
Most
common
types
of
applications
that
are
powered
by
flink
are
even
driven
data
analytics
and
data
pipeline
applications.
B
I'll
just
take
an
example.
If
you
are
streaming
a
game
or
something
right,
you
can
use
fling
connect
flink
for
to
do
that.
So
in
the
road
map.
There's
an
effort
around
building
a
flink
delta
source
that
is
flink
reads
from
delta
lake,
using
the
delta
standalone
reader
and
with
the
upcoming
delta
standalone
rider.
It
can
allow
flink
to
drive
to
delta
tables
and
there's
an
effort
around
a
delta
sync
connector
for
flick.
B
B
Second,
one
is
pulsar,
so
pulsar
is
an
open
source
streaming
project
that
was
built
at
yahoo
as
a
streaming
platform
in
terms
of
delta.
We
are
bringing
streaming
enhancements
to
standalone
reader
to
support
pulsar.
There
are
two
connectors
which
are
being
worked
on
one
for
one
for
reading
from
delta
table
as
a
source
and
another
is
writing
to
the
delta
table
as
a
sync.
B
C
We're
pretty
active
there.
We'd
love
to
listen
to
and
get
your
feedback
so
and
later
on.
We'll
also
be
asking
for
your
feedback
from
a
survey
as
well
too
so,
but
yes
absolutely
join
us
here,
because
it's
a
very
active
project
right
now-
and
I
believe
we
call
this
out
in
a
few
slides
from
now,
but
we're
targeting
at
basically
around
q3
q4
time
frame
to
be
at
least
to
have
a
preview
of
this
code
base.
B
Awesome
awesome
and
for
those
who
don't
know
what
trino
is
it's
an
ansi,
sql
compliant
query
engine
off
of
presto
that
works
with
bi
tools
such
as
r
tableau
power,
bi,
supersite
and
so
on
and
yeah.
So,
as
danny
mentioned,
chino
and
presto
dv
are
actively
being
worked
on
coming
from
the
same
ecosystem.
B
Presto
is
an
open
source
distributed
sql
query
engine
for
running
interactive,
analytic
queries,
so
we
are
also
working
on
getting
a
presto
delta
reader
to
allow
press
store
to
read
from
delta
tables,
so
the
channel
for
that
is
connector
presto
and
you
can
join
our
bi-weekly
meetings
on
thursdays
and
then
rust.
So
it
gets
me
excited
for
those
who
are
new
to
rust.
Rust
is
a
programming
language
that
helps
you
write
faster,
more
reliable
software.
B
B
B
What
it
will
do,
basically,
is
like
if
you
have
data
coming
from
source
systems
through
kafka
and
it
is
being
ingested
into
delta
lake.
In
the
rest,
you
can
write
the
whole
workflow
and
you
just
don't
need
to
have
like
manual
configurations
done.
You
can
just
do
it
in
code
and
it
simplifies
the
operation.
B
It
enables
you
to
manage
your
data
lake.
The
way
you
manage
your
code,
so
we
are
working
on
a
very
tight
integration
between
league,
fs
and
delta
to
allow
for
experience
for
data
lakes
to
work
with
delta
lake.
For
this
the
channel
is
lakefs.
You
can
join
that
channel
or
we
also
have
bi-weekly
meetings
coming
up
soon.
We
will
post
it
there
and
then
hive
to
delta
connector.
B
and
then
apache
spark
right.
We
have
seen
great
days
of
innovation
in
spark
and
with
that,
as
I
mentioned
before,
we
want
to
keep
the
innovation
happen
with
the
pace
of
innovation
at
spark.
So
there
are
two
main
things
coming
up
in
the
roadmap.
We
want
to
provide
support
for
what
spark
is
offering,
which
is
support,
column
drop
and
rename
commands
and
then
support
apaches
part
3.2.
B
So
there
were,
there
were
like
exciting
announcements
on
connectors,
so
putting
it
all.
Together,
we
reviewed
how
the
delta
community
is
rapidly
expanding
and
we
have
connectors
planned
in
the
roadmap.
Here's
the
screenshot,
where
you
can
see
when
when
is
the
connector
coming
and
if
you
don't
see
any
connector
in
the
list,
if
you
have
any
requests,
please
let
us
know
okay,
so
another
project
underway
that
I
want
to
talk
about
gets
me
excited.
Is
delta
sharing,
there's
a
growing
demand
for
ways
to
share
data,
that
is
beyond
the
walls
of
the
organization.
B
For
example,
what
if
you
want
to
share
your
data
with
external
entities
like
partners,
customers?
What
not
users
sometimes
are
frustrated
with
the
constraints
on
how
they
can
share
their
data?
Think
csv
files
file
size
limits
and
once
that
data
is
shared
version,
control
and
data
freshness
are
so
tricky
to
maintain.
B
B
They
as
they
suddenly
realize
that
they
need
another
data
set,
so
they
submit
a
ticket
and
they
wait
for
maybe
like
few
weeks
so
in
the
two
weeks
or
so
it
takes
them
to
it.
It
takes
the
it
it
takes
a
you
know,
rhythm
out.
It
takes
so
long
to
get
the
data
access,
so
the
time
is
lost
conditions
change
and
they
lose
the
momentum
of
that
insight.
B
Data
sharing
shouldn't
be
a
barrier
to
innovation.
This
is
why
I'm
excited
to
talk
about
delta
sharing,
which
is
industry's
first
open
protocol
for
secure
data
sharing,
making
it
simple
to
share
the
data
with
organization,
regardless
of
what
compute
platforms
they
use
and
to
illustrate
how
this
works.
Let's
look
and
look
at
under
the
hood,
so
let's
consider
this
example
of
data
recipient
getting
most
recent
sales
data
from
a
data
provider.
B
Permissions
for
the
authentication
credentials
are
then
checked
against
access
granted
in
the
delta
sharing
server.
These
access
permissions
are
then
translated
by
delta
lake
to
identify
if
the
pertinent
underlying
parking
files
exist
and
it
identifies
it
since
permissions
may
be
specific
to
a
subset
of
the
table.
Delta
lake
applies
the
permission
to
find
only
those
files
which
the
authenticated
credentials
have
access
to
see.
B
The
data
recipient
uses
these
signed
urls
to
directly
access
the
data
files
from
their
cloud
storage,
location
and
this
delta
sharing
uses
an
open
format
so
that
it
can
allow
external
programs
and
clients
to
access
data
with
delta
sharing.
Not
only
you
can
share
data,
but
you
can
also
share
machine
learning
models
as
assets
with
other
users
pretty
exciting.
B
B
We
are
also
seeing
many
of
the
leading
commercial
vendors
in
data
management,
business,
intelligence
and
analytics
vendors,
as
well
as
digital
native
companies,
supporting
this
as
well
with
that.
I
want
to
highlight
this
quote
from
one
of
our
project:
committers
and
now
a
beacon
karen
will
share
more
about
the
beacon
who
says,
with
delta
leak,
1.0
delta.
Now
is
ready
for
every
workload.
With
this
message,
I
want
to
emphasize
the
importance
of
having
these
amazing
contributors
without
whom
this
journey
would
not
be
easy.
This
is
why
we
always
look
for
community
participation.
B
If
you
haven't
already
been,
there
are
a
bunch
of
channels
on
how
you
get
involved
with
us
check
us
out
on
slack
check
us
out
on
youtube
channel
where
you
can
find
this
video
or
another
video
there's
a
mailing
list.
If
you
want
to
get
involved
with
the
actual
code,
you
can
also
join
us
on
github
and
more
and
yeah.
I
would
like
to
call
out
the
survey:
do
you
wanna
kick
this
off
denny.
C
Absolutely
so
so
for
everybody.
Thank
you.
Vinnie
you've
done
an
amazing
job
by
the
way
for
all
the
folks
who
are
asking
questions,
keep
on
asking
them
and
we'll
probably
answer
some
of
them
live
anyways
because
they're
actually
worthwhile
for
us
to
answer
live,
but
just
keep
on
answering
questions
just
because
we
know
we've
packed
a
lot
of
information.
So
thank
you
very
much
vinnie
now,
specifically
to
the
roadmap
survey.
C
The
implication
of
what
vinnie's
talked
about
is
exactly
that.
There's
a
lot
of
things
going
on
with
integration
right,
there's
again,
whether
it's
lake
fest
or
nessie
or
trino
preston.
So
we
would
love
to
get
your
feedback
on
that
we'd
love
to
understand
better
from
you,
the
community
delta
lake
community.
C
What
is
important
to
you
and
how
should
we
prioritize
these
things?
We
do
this
at
a
roughly
six-month
cadence,
so
it
doesn't
mean
we
don't
listen
to
feedback
by
the
way
between
the
six
months.
We
absolutely
do
we
just
we
just
sort
of
formalize
it
every
six
months,
and
also
honestly,
I
don't
think
my
management's
gonna.
Let
me
give
out
t-shirts
more
often
than
every
six
months.
Okay,
so
just
shout
out
to
ryan,
I'm
probably
in
a
lot
of
trouble
already
residents,
so
please
do
fill
out
the
survey
the
link's
there.
C
The
link
has
also
been
submitted
to
youtube,
zoom
and
linkedin,
so
you
all
can
chime
in
there.
So
you
don't
have
to
copy
paste
that
if
you've
already
filled
out
the
survey-
and
I'm
gonna
blame
myself
on
this
one-
that's
why
vinnie
made
sure
I
actually
had
to
go
ahead
and
call
this
out.
Can
you
go
ahead
and
log
back
in
and
fill
out
the
survey
again?
Not
all
of
it.
C
We've
kept
everything,
but
I
forgot
to
ask
for
your
address
in
order
to
send
your
t-shirt,
so
my
bad
on
that
one.
So
if
you've
already
filled
out
the
survey,
please
log
back
in
and
add
in
your
address
information,
if
you
still
want
the
t-shirt,
if
you
don't
but
did
indicate,
we
will
email
you
for
that
information
anyways,
but
just
to
simplify
things.
If
you
can
do
that
so
yeah,
my
bad
on
that
one,
but
please
this
survey
will
be
open
at
least
till.
C
So
so
please
do
we
again
you
get
a
free
t-shirt
at
a
minimum
and
also
get
a
joint
in
a
raffle
for
the
edition
of
the
o'reilly
defending
guide
delta
lake
delta
lakes,
the
definitive
guy
from
o'reilly
by
the
way
two
of
the
co-authors
you're
talking
to
them
right
now,
so
there
you
go
so
that
was
also
just
a
a
little
bit
of
a
call
out
for
us
on
a
shameful
blog.
Basically,
all
right,
that's
it
for
the
roadmap
survey.
Thank
you.
B
C
Well,
in
terms
of
questions,
basically
how
about
this
when
I
go
ahead
and
chime
to
some
of
the
questions
from
linkedin
for
starters?
Okay,
yes,
okay,
perfect!
So
actually
mike
from
linkedin,
actually
had
an
excellent
question,
which
is
basically,
if
you're,
implementing
delta
lake
on
an
existing
data
lake.
Is
there
an
impact
on
data
flow
speeders
and
not
noticeable
at
all
and
so
I'll?
C
If
it's
okay,
I'm
going
to
take
this
and
I'll
answer
this
particular
question,
okay
and
so
mike
for
starters,
excellent
question:
I
did
actually
respond
to
linkedin
already,
but
I
figured
I
can
also
add
a
little
bit
of
color
to
this.
So,
first
of
all,
when
you're
talking
about
delta
lake,
it
is
a
storage
layer,
so
there's
not
a
processing
engine
that
goes
with
it.
C
So
it's
not
like
there's
anything
inherently
about
a
storage
layer
that
allows
us
to
go
ahead
and
speed
things
up
on
a
on
a
separate
decoupled
processing
engine
saying
that
ultimately,
there's
a
lot
of
things
that
have
to
do
with
the
file
system.
So,
with
the
way
delta
lake
works
is
that
we
can
do
things
to
structure
the
file
system,
so
that
way
the
process
engine,
whether
it's
spark
or
any
of
these
other
systems
that
we
were
talking
about
here
today.
They
can
go
ahead
and
read
these
things
significantly
faster.
C
Now,
in
the
case
of,
for
example,
I'm
going
to
use
one
simple
example:
when
you're
using
cloud
object,
stores,
there's
a
lot
of
files,
the
listing
of
those
files
actually
is
extremely
slow.
You
know,
for
example,
if
you're
writing
in
your
computer,
doing
fs
you're
doing
an
ls60
listing
in
the
files
on
your
local
computer
with
an
ssd.
That's
actually
really
fast.
C
On
a
cloud
object
start,
that's
actually
quite
slow.
So
in
that
particular
case
you
know
that
actually
would
be
extremely
non-performant,
extreme,
very
slow
in
the
case
of
delta
lake.
That
information
is
actually
stored
in
the
transaction
log
itself,
the
metadata
there,
and
because
it
has
that
it's
actually
able
to
read
the
list
of
files
directly
from
the
transaction
log,
and
then
your
distributed
system
like
spark,
can
go
ahead
and
actually
send
out
multiple
tasks
to
go.
C
Read
for
that
data,
and
so
things
like
this
help
optimize
the
performance
of
your
system,
so
so
it
can
be
quite
noticeable
now,
in
the
case
of
open
source
delta
lake,
for
example,
I'm
going
to
call
ourselves
out,
there's
been
asked
for
us
to
implement
an
open
source
version
of
optimize.
Okay.
So
that's
not
currently
on
the
roadmap,
but
if
you
look
join
the
github
issues,
that's
what
people
are
asking
so
right
now
we
are
currently
examining.
C
What
are
the
right
approaches
for
us
to
do
that
optimize
currently
is
a
databricks
feature,
not
an
oss
feature,
but
we've
been
again,
we
listen
to
the
community
and
we
are
trying
to
figure
out
what's
the
right
way
and
the
timing
for
us
to
do
that.
Optimizes
allows
us
to
ultimately,
though,
restructure
the
files
in
a
slightly
different
way
to
basically
go
ahead
and
improve
the
performance
that
even
more
so
not
so
much
because
of
again
we're
not
affecting
you
know,
presto
or
trino
or
or
or
flink.
C
When
you
know
or
spark
you
know
we're
not
affecting
those
engines,
but
the
structure
of
the
data
is
optimizing
more
with
additional
indexes
and
things
of
that
nature
that
will
ultimately
help
improve
performance.
So
hopefully
that
answers
your
question
and
hopefully
to
provide
context.
Hopefully
that
answers
provide
a
little
context
on
why
it's
so
important
to
actually
have
something
like
delta
lake,
in
order
to
not
just
ensure
the
reliability
of
your
data,
but
also
improve
the
performance
of
your
data.
C
B
B
B
C
I
also
was
on
the
team
that
built
what
ultimately
was
now
known
as
hc
insight,
hadoop
on
windows
who
took
on
azure,
then
obviously
we
switched
over
linux
because
that's
obviously
a
smart
thing
to
do.
But
the
reason
I'm
calling
that
out
is
because,
when
you're
talking
about
databases
like
sql
server,
they
they
are
a
single
system
that
are
hard
to
distribute
and
ultimately
you
could
try
to
distribute
them
by
basically
sharding
the
database.
C
But
that
becomes
a
management
overhead
that
becomes
excessively
complex,
complicated
but
databases
into
their
pro
have
acid
transactions
right,
and
so,
in
other
words,
they
ensure
the
reliability
or
data
with
the
v2.
That's
the
v1
of
the
data
engineer
where
we
started
with
databases.
The
v2
of
the
data
engineering
world
is
for
us
to
say
things
like.
Okay.
Let's
just
do
data
links
and-
and
I
was
guilty
of
that
by
the
way
you
know
it's
like
we
just
shove
everything
and
as
vinnie
alluded
to
it's
like
it's
it
very
much.
Is
this
concept
of
great.
C
We
store
everything
but
garbage
in
garbage
out.
Okay,
so
you,
if
you
store
garbage
you're,
going
to
get
garbage
right,
and
so
why?
Because
it
was
lacking
that
reliability
was
lacking.
The
idea
of
acid
transactions
schema
enforcement
things
to
ensure
that
the
data
actually
was
good,
which
is
what
we
cared
about
in
v1,
okay,
but
v2.
We
wanted
that
flexibility.
So
that's
why
you
know
hadoop
and
data
lakes
in
general
became
so
prominent.
So
now
v3
is
of
the
data
engineering,
world
society
of
a
lake
house
right
the
best
of
both
worlds
right.
C
We
constantly
do
these
things
best
of
both
worlds,
right
and
there's,
I'm
sure,
there's
a
star
trek
particular
theme
that
I
could
throw
on
there
I'll.
I
won't
digress
too
much,
but
the
context
is
we
want
to
take
the
acid
transactions
and
that
reliability
of
a
database,
but
also
give
us
that
flexibility
with
the
data
lake
and
then
that
allows
us
to
build
that
like
house
that
you
so
desperately
need
okay,
because
in
this
world,
as
we
progress
and
that
build
have
more
and
more
data.
Ultimately,
that's.
C
What's
that,
that's
what
you
actually
need!
You
need
the
melding
of
those
both
worlds.
So
as
a
sql
server
guy,
you
know,
like
I
said
I
was
part
of
this
equal
surrender
dean
right
as
much
as
I
love
that
database.
The
the
the
this
new
world
really
does
require
us
to
shift
over.
I
don't
I'm
not
trying
to
bash
sql
server.
Quite
the
opposite.
C
There's
still
many
scenarios
where
databases
themselves
are
extremely
useful
and
you
should
use
them,
but
when
it
comes
to
the
the
flexibility
and
the
sheer
volume
and
the
fact
that
you're
trying
to
do
machine
learning
data
science,
ai
streaming,
all
of
these
other
data
domains
all
at
the
same
time.
That's
the
reason
why
it's
so
important
to
make
that
shift
to
the
lake
house
and
fundamentally
delta
lake
is
required
for
that
lake
house
to
work
I'll
get
off
myself
on
staff.
C
Thank
you.
Thank
you.
Let's
see
we,
the
next
question
is
is
evil.
Unfortunately,
because
it's
short,
do
we
plan
to
put
s
any
sap
connectors
in
the
future
for
delta
lake?
We
don't
have
any
plans
for
that
right
now,
but
that
doesn't
mean
we
can't
do
it
by
the
way.
In
other
words,
this
is
why
we
want
community
feedback.
So
while
I'm
giving
you
the
answer,
we
don't
have
any
feedback
on
that.
C
If
you
chime
in
in
the
survey
or
if
you
chime
in
or
so
that's
a
delta
user
slack,
and
we
get
enough
feedback
from
the
community
that
you're
interested
in
working
with
sap
connectors
and
then
exactly
what
type
of
sap
connectors
you're
talking
about.
You're
talking
about
hana
you're
talking
about
the
erp
systems,
what
what
what's
the
nature
of
those
connectors
then
yeah?
It's
not
like
we're
against
the
idea
of
doing
it.
C
B
A
C
Okay,
I
think
the
question
here
is
basically
yeah,
basically
adding
column
stats
right
now.
We
don't
have
plans
to
do
it
yet,
just
because
of
the
nature
of
the
way
the
column
stats
works
within
the
data
bricks
runtime,
if
it's,
if
there's
a
concern
from
a
performance
perspective,
definitely
chat
with
us,
because
in
fact
we
have
very
good
details
and
evidence
of
the
fact
that
the
performance
even
on
delta
oss
is
actually
quite
fast.
Without
those
column
stacks.
It
doesn't
mean
we're
not
interested
in
doing
by
the
way.
C
B
Yeah-
and
one
more
thing
I
would
add,
is
like
collecting:
if
you
have
like
long
strings
or
something
collecting,
statistics
is
expensive
operations,
so
you
can
either
configure
the
table
properties.
You
know
to
avoid
columns
containing
long
strings,
but
yeah
eventually
like.
If
you
have
any
requests
like
this,
we
would
love
to
get
it
on
the
survey.
C
Oh
perfect,
there's
a
great
question
that
just
followed
that
just
came
from
kosher
from
linkedin,
so
what
does
it
take
to
realize
an
idea
from
the
community
to
actually
put
it
on
the
roadmap?
Great
question?
Okay,
so
here's
exactly
the
approach
and
that
that
you
want
to
go
do
number
one
obviously
fill
out
the
survey
and
and
work
with
the
community,
whether
we're
talking
to
the
delta
user,
slack
channel
or
the
delta
users
distribution
list
saying
that?
Okay,
that's
obviously
you
want
to
get
involved.
C
Don't
forget.
The
other
key
things
is
is
to
actually
create
github
issues.
Okay
and
chime
in
that
these
are
things
that
you
want
to
see
the
re.
The
reason.
For
example,
we
chose
the
the
issues
that
listed.
You
see
like
right
now
that
vinnie
just
presented
it
wasn't
because
we
just
randomly
chose
them.
C
That's
why,
for
example,
there's
this
feedback
of
a
preston
trina,
but
that's
also
part
of
the
reason
why
we
created
the
survey
which
is
like
we
knew
we
were
getting
feedback
directly,
that
wasn't
necessarily
from
github,
but
we
won
that
before
we
did
all
these
things,
we
wanted
to
go
ahead
and
chime
out
and
ask
the
community
for
their
their
input.
So
that
way
we
can
figure
out
how
to
prioritize
accordingly.
C
So
if
we
don't
want
to
go
through
that
route,
yeah
create
a
github
issue
and
then
get
other
community
members
to
chime
in
to
vote
for
to
actually
say
yeah.
We
see
that
this
is
important
and
then
that's
actually
how
you
get
it
on
the
roadmap,
because
this
this
is
a
community
effort.
So
really
definitely
join
us
in
the
in
the
in
the
the
the
delta
lake.
C
Again,
the
slack
channel
is
probably
the
best
way
because
we're
a
lot
better
about
answering
that
more
than
else,
but
whether
it's
that
or
the
user
groups
they're
all
the
same.
Please
go
ahead
and
chime
in
and
as
vinnie
also
called
out
for
a
lot
of
these
integration
efforts.
For
example,
we
have
bi-weekly
meetings
as
well.
So
if
he's
chiming
on
there
too,
so
lots
of
really
good
stuff
for
you
to
to
provide
your
input,
provide
your
feedback
and,
of
course,
we're
always
looking
for
contributors.
A
B
Yeah
and
we
regularly
post
updates,
so
you
can
follow
danny
me
and
karen
on
linkedin
or
twitter,
so
you
can
get
exciting
updates
and
also
like
if
you
want
to
have
hands-on
coding
and
contribute
to
our
code
base.
You
can
follow
us
on
github,
so
we
regularly
have
active
discussions
going
on.
So
please
do
that
and
you
can
get
to
know
what
features
we
are
releasing
before.
We
even
call
a
bigger
event.
A
A
We
kind
of
have
to
tweak
it
and
see
see
what
would
make
sense,
maybe
an
hour
or
so
session,
but
the
polls,
just
if
you're
interested,
if
you're
interested,
we
can
send
you
more
information
when
we
kind
of
flush
out
the
idea,
a
little
bit
more
so
feel
free
to
post
in
the
chat
too,
on
linkedin
on
youtube.
If
you'd,
if
that's
something
you'd
be
interested
in.
B
C
Yeah
I
was
about
to
chime
in
looks
like
since
we're
wrapping
up
right
now
with
the
poll
and
all
these
things,
our
apologies
to
everybody
who
we
have
not
been
able
to
answer
questions.
It's
not
lack
of
interest.
It's
just
lack
of
time.
I
guess
so.
Please
do
join
us
on
the
delta
lake
users
slack
channel.
We
are
whether
it's
vinnie
and
myself,
but
also
various
other
engineers,
we're
all
there
to
answer
your
questions
and
so
obviously
we're
doing
asynchronously.
C
So
we
may
not
be
able
to
get
to
right
the
second,
but
the
point
is
that
we
definitely
are
there
to
answer
questions.
So
please
be
be
comfortable
to
answer.
Ask
your
questions
there.
We
have
lots
of
different
channels
to
chime
in
and
for
that
matter
you
can
always
tag
me
and
I'll
do
my
best
to
answer
your
questions
as
soon
as
I
get
enough
coffee
in
my
system,
so
there
you
go.
B
There's
another
question
from
chris:
I
think
it's
along
the
similar
lines
of
how
people
can
get
engaged.
If
they
are,
they
haven't,
contributed
to
open
source
previously.
So
what
we
will
do
is
we
will
share
the
slides
or
this
recordings
are
available
and
there
are
different
ways
you
can
join
us
either
on
github
or
maybe
you
can
join
us
through
different
channels
for
different
projects,
and
we
can
show
how
you
can
engage
with
us.
A
Awesome
so
I
think
I
think,
with
that
it
looks
like
about
50
of
the
folks
in
zoom
have
answered
my
poll
question.
So
that's
awesome
looks
like
most
folks
would
be
interested
in
a
hands-on
workshop,
so
we'll
we'll
go
ahead
and
start
planning,
something
which
I
think
will
be
cool.
We
haven't
done
a
workshop
kind
of
style
online
event
before
so
that'll
be
cool
to
to
do
one
about
delta
lake.
A
So
awesome,
thank
you,
everyone,
so
much
for
joining
and
for
all
of
your
questions
and
thank
you
so
much
vinnie
loved
hearing
about
what's
coming
up,
excuse
me
and
of
course,
denny
always
a
pleasure.
So
with
that
everyone
I
hope
you
have
a
great
rest
of
your
day
and
take
care
yeah.
Thank
you.
All
bye.