►
From YouTube: Delta Hack: Committers AMA and Happy Hour
Description
Join us for a casual Ask Me Anything and happy hour with some of the Delta Lake committers. Let's talk about the future of Delta Lake, fun problems we're going to solve, or cool uses of Delta Lake!
B
Boom
live
streaming
is
enabled
that
means
we
are
live
with
a
very
serious,
very
serious
session,
with
we've
got
ryan
and
denny.
I'm
waiting
to
see
if
qp
will
join
us
for
a
committer's,
ama
and
happy
hour.
I
was
just
lamenting
to
denny
that
I
don't
have
any
beer,
so
my
happy
hour
is
a
little
less
happy,
but
it
will.
I
have
I
have
a
water
here,
so
it
will
be
hydrating.
B
A
C
B
C
So
basically,
like
I
basic
star
work
with
like
michael
who
is
the
likely
author
of
the
data
lake,
which
basically
ended
up
at
the
beginning.
So
I
made
like
a
lot
of
commits
to
the
toyota
lake
like
in
the
past
several
years
and
in
addition,
I'm
also
like
a
spark
update
bar
committee,
which
also
contributes
like
hundreds
of
commits
to
our
budget
spark
in
the
past
yeah
and
then
now
we
are,
I'm
basically
focused
on
like
teletubbing,
which
is
a
new
open
source
project.
B
B
C
B
So
let
me
let
me
ask
you
ryan,
with
delta
sharing,
I
mean
a
lot
of
it.
A
lot
of
it
was
coming
together,
sort
of
in
the
in
the
final
final
weeks
before
data
nai
summit,
where,
where
y'all
announced
it
yeah
were
there,
were
there
any
features
or
any
things
that
you
you
had
to
like
cut
to
make
the
deadline
that
you
you're
you're,
looking
forward
to
bringing
in
now
that
data
and
ai
summit
is
over.
C
Yeah
so
basically
yeah,
because
we
actually
just
had
like
only
several
weeks
before
they
don't
have
time.
So,
basically,
we
are
just.
We
were
basically
focusing
on
like
the
major
features
like
python
connector
and
also
like
a
spark
connector
that
you
can
read
from
this
like
a
data
sharing
server
and
then
the
open
source
data,
australian
server.
C
Kindly
right
now
is
kind
of
a
reference
implementation
which
supports
the
major
features,
but
regarding
like
a
security
or
other
like
optimizations,
we
actually
haven't
spent
quite
a
lot
on
these
parts
and
yeah
for
the
open
source
client.
There
are
also
like
a
lot
of
peace.
We
can
improve
in
the
future
like
a
currently
we
are.
C
A
B
B
I'm
I'm
interested
ryan
to
know
like
where
you
imagine
this
going
like
delta
delta
itself.
Has
all
of
these
features
like
time
travel.
There's
right,
I
mean
there's
writer
support
as
well,
and
delta.
Sharing
right
now
is
very
reader
oriented,
but
it's
sort
of
like
a
single
snapshot
in
time
of
of
a
data
set,
how
much
of
like
what
we
see
in
delta,
as
sort
of
like
support
or
or
compatibility
do,
you
think,
is
going
to
come
over
into
delta
sharing.
C
So
you
so
maybe
we
can
use
some
example
like
time
travel
like
you
mentioned,
let's,
let's
definitely
we
can
support
this
easily
in
this
internationally.
We
can
extend
a
kind
of
protocol
like
you
can
specify
a
table
version
in
a
request,
and
then
we
can
support
this
and
and
another,
like
other
features
like
kindly.
C
B
B
C
C
B
B
C
A
C
B
So
qp
we
were
just
talking
about
about
delta
sharing
and
that
sort
of
led
us
into
some
of
the
concurrency
fund
that
we've
had
in
delta
rs.
Since
you
joined,
I
want
to
make
use
of
your
time.
Let
me
ask
you
a
question
you
know
for,
for
anybody
that
that
doesn't
know
who
qp
is
qp
is
the
original
creator
and
committer
on
delta
rs,
which
he
created
last
year.
I'm
kind
of
curious
where
you
think
delta
rs
goes
over
the
next.
Let's
say
six
to
12
months.
D
So
obviously
the
most
important
thing
is
to
get
into
production
a
script,
so
we
have
a
pretty
cool
project
called
delta
interest.
I
think
that
that's
the
first
goal
that
I
would
like
to
see
happens
and
then,
after
that,
I
think
there
are
a
couple
interesting
use
use
cases
that
could
come
out
of
it.
For
example,
we
have
been
talking
about
well,
we've
been
we'll
say,
tighter
I've
been
talking
about
using
lambda
to
do
some
cool
delta
liquidated
hacking.
D
B
It's
I
I
feel
like
sometimes
with
these
these
ideas
that
I'll
have,
I
feel,
like
I'm
a
toddler
who's
like.
Why
can't
we
do
this,
but
what
about
that?
What
if
we
did
this,
would
that
work
and
then
kp's
like
well
yeah,
maybe-
and
I
feel,
like
I'm
spending,
far
less
intellectual
time.
Thinking
about
how
possible
it
is,
I'm
just
like
what,
if
we
did
it
like
this
yep.
D
D
B
D
Efficient,
it's
basically
a
query
execution
engine
and
an
analytical
query
execution
engine
based
on
arrow,
the
arrow
in
memory
format
and
it's
sort
of
similar
to
pandas
for
python.
But
it's
for
rust
or
or
in
rust,
and
it
implements
the
same
set
of
data
frame
api
and
then,
on
top
of
that
it
provides
a
sql
query
interface
for
you
to
query
any
kind
of
tabular
data
or
that
that
in
any
language
well
in
rust.
But
I
think
the
cool
thing
about
that
is
because
it's
in
rust.
B
C
A
B
A
A
Okay,
fair
enough.
Well,
then,
for
me
it's
we've
got
a
lot
of
people
who
are
asking
or
interested
in
various
connectors,
like
you
know,
flink
presto,
whatever
else,
and
so
we've
actually
even
started
various
channels
on
the
delta
lake
slack
channel
to
go
ahead
and
talk
about
that,
and
in
fact,
that's
probably
what
I
would
love
to
have
more
involvement
within.
You
know
the
flint
community,
the
presto
community
kafka
communities,
all
these
other
communities
that
really
want
to
it's,
not
just
about
them.
Telling
us
hey,
we'd,
love
to
see
it.
A
It's
like
no
we'd
love
to
go
ahead
and
actually
sit
down
and
engage
more,
and
so
it's
starting
to
happen
like
I
said
so,
we've
started
placing
some
channels
in
delta
lake
specifically
for
that,
but
love
to
see
you
know,
since
we've
got
a
nice,
you
know
promotion
vehicle
here
right
now,
right,
love
to
actually
have
more
people
involved,
we'll
be
starting
once
every
two
week
meetings
I
believe
in
I
want
to
say
early
july.
That's
when
we're
starting
those
meetings.
Yeah,
I
think
the
flink
one
we've
already.
A
A
D
B
All
right,
I'm
gonna,
have
to
go
dig
up
that
issue
now,
one
of
our
good
first
issues,
one
of
the
things
I'll
I'll
just
call
out
here.
There's
a
lot
of
good.
First
issues
linked
for
delta,
rs
ryan
have
y'all
done
that
for
the
delta
repo
as
well
gone
and
tagged
issues
that
are
like
beginner
or
you
know,
introductory
things.
B
A
B
A
B
Week,
let
me
go
back
to
my
my
list
of
questions
again.
If
you've
got
any
questions,
you're
welcome
to
throw
them
in
the
youtube
chat
and
we'll
see
what
we
can
do
to
to
answer
them,
maybe
denny's
something
that
you
might
be
willing
to
share,
or
maybe
not
we'll
see,
but
I'm
curious,
let's
say
over
the
past
year
or
so
like.
What's
the
coolest
use
of
delta
lake,
that
you've
seen
that
you
are
able
to
talk
about
publicly.
A
I
think
the
coolest
use
in
some
ways
is
actually
almost
pretty
boring
right.
So
the
reason
I
say
that
is
because
a
lot
of
this
has
to
do
with
the
the
I
can't
mention
the
company,
but
I
can
definitely
mention
the
use
case,
and
so
the
use
case
actually
is
very
much
dpr
and
in
fact,
if
you,
if
you
do
a
bunch
of
youtube
stuff,
you
probably
will
find
the
company.
What
was
that
acronym?
Oh
sorry,
gdpr
compliance.
B
A
Compliance
actually
wasn't
just
deleting
the
information
but
was
actually
updating
the
information
first,
because
that
way
they
can
ensure
that
their
downstream
systems
could
also
comply
properly
with
ccp
and
gtbr,
and
then
finally,
they
would
delete
afterwards.
But
the
context
was
that
first
was
an
update,
and
that
would
signal-
and
that
would
trigger
a
bunch
of
downstream
systems
and
in
the
process
of
doing
that,
because
people
are
saying.
Oh,
how
can
you
do
update
end
up
involving
a
bunch
of
data
warehousing
techniques,
so
in
other
words,
your
surrogate
keys,
slowly
changing
dimensions?
A
You
know
type
2,
slowly,
changing
dimensions.
A
lot
of
the
traditional
data.
Warehousing
type
techniques
are
actually
basically
applied
with
the
delta
link,
and
so
that
part
I
find
sort
of
cool
just
because
I'm
an
old
data
warehousing
hound
and
the
idea
that
my
old
data
warehousing
knowledge
actually
is
applicable
to
the
new
world.
Today
it's
actually
pretty
sweet,
so.
B
A
B
C
Yeah
one
thing
like,
for
example,
in
the
past
we
saw
like
some
customers
that
have
like
very
like
big
tables
and
then
before
delta.
Actually,
a
query
is
very
slow
because
when
you
are
creating
this
metadata,
if
you
need
to
like
read
from
high
mega
store-
and
unfortunately
it's
like
from
like
a
database-
is
pretty
slow
and
then
after
we
like
this,
let's
switch
it
to
delta
lag.
D
D
C
C
D
So
I
I
I've
kept
going
back
to
kafka
down
the
stream
because
big
interest,
because
I
think
we
are
also
sort
of
stress
testing
the
transaction
action
in
delta
particle,
where
I
think
we
might
be
the
first
application
that
try
to
use
try
to
create
one
transaction
application
per
per
copy
partition
to
keep
track
of
the
ingestion
progress
within
the
other
table
itself.
So
we'll
see
how
it
goes.
Hopefully,
it'll
go
well,
but
so
far
we're
pretty
happy
with
the
design.
B
A
A
Like
you
know,
we
definitely
started
with
spark
right,
and
so,
let's
not
bother
the
pretense
that
we
haven't
done
that
right,
but
the
reason
we
wanted
the
connectors
is
because
interacting
with
delta
lake
and
utilizing
the
advantages
of
asset
transactions
for
your
data,
lake
with
delta
lake
is
not
specifically
a
spark
project
right.
It's.
It
can
be
anything
right.
It
doesn't
matter
what
language
you're
using
doesn't
matter.
A
A
There
are
pull
requests
for
the
presto
connector
right
now
that
is
currently
being
worked
on
the
obviously
the
stuff
that
ryan
has
created.
The
hive
connectors
jvm
standalone,
the
delta
native
standalone
reader,
things
of
that
nature.
So
we'd
like
this
is
just
one
repo,
though
right.
We
have
actually
have
many
connectors
that
can
talk
to
delta
and
both
from
right
now,
mostly
primarily
on
read,
but
we're
actually
working
closely
with
the
community,
including
teams
like
yourselves
for
also
writing
capacity.
A
B
C
In
this
case,
we
like
to
bump
at
least
write
a
version,
because
the
other
version
will
be
able
to
understand
how
to
generate
these
values,
just
based
on
this
expression
for
the
later
version
part
which
we
haven't
done
any
change
in
the
past,
but
in
future
we
may
like
bomb
police,
for
example
like
if
we
would
like
to
support
like
a
hidden
columns
that,
like
we
would
like
to
allow
the
user
space
file.
Okay,
I
don't
want
to
show
this
column
in
a
table.
I
need
just
some
metadata
something
like
this.
C
B
C
What
do
you
think
right
yeah?
So,
basically
we,
if
you
like
to
change
the
protocol
and
first
I
think
it's
better
to
like
discuss
in
a
delta
core
project,
which
you
may
just
open
a
pool
request,
a
change
to
the
protocol,
and
then
we
can
discuss
later
what
you
like
to
change,
and
how
can
we
coordinate
this
with
other
projects.
D
C
D
B
Six
to
12
months,
because
we
have,
we
have
a
lot
of
versions
of
things
now
yeah
there
there's
the
spark
implementation
version,
there's
the
the
standalone
reader
and
then
the
delta
rs
has
its
own
versioning
and
we
don't
have
any
shared
understanding.
Right
now
of
you
know.
Delta
lake
is
1.0.
We
said
that
at
data
in
nai
summit.
C
A
So,
in
addition
to
the
document,
I
feel
that
that
that's
a
good
way
for
us
to
make
sure
everybody's
on
the
same
page
that
we
actually
just
just
have
more
discussions
on
this,
because
there's
also
different
interpretations
of
what
the
protocol
is
doing.
Right
we've
been
talking
to
plenty
of
just
like
you
guys
right.
We've
been
talking
to
plenty
of
other
partners
about
how
they
want
to
use
delta,
and
you
know
what
ultimately
happens
like
there
was
a
different,
even
though
we're
all
reading
the
same
boards.
It
was
a
different
based
on
the
context.
A
B
D
B
C
C
B
Almost
we're
delta
0.99
compatible,
not
quite
not
quite
there,
so
just
to
be
respectful
of
of
ryan,
denny
and
qp's
time.
I
could
talk
all
day,
but
let's
give
it
a
a
a
couple
more
minutes.
If
you've
got
any
questions,
please
dump
them
in
the
in
the
youtube
chat.
If
you've
got
a
question,
if
it's,
this
is
no
longer
live
and
you're
watching
this
through
through
youtube,
you
can
go
to
delta.io
and
scroll
to
the
bottom.
There's
the
the
link
to
the
slack
workspace.
B
A
A
A
D
So
I'm
planning
to
so
previously
tobacco.
On
this,
we
we
have
we're
working
on
this
multi
writer
for
s3
support,
right,
which
you
all
were
talking
about
about
lack
of
access
to
atomic
rename
industry.
So
we
have
a
design
that
we
sort
of
partially
improved
with
tla
plus
and
during
that
process
I
found
a
lot
of
problem.
It's
actually
not
a
really
easy
to
think
easy
thing
to
use.
D
So
I'm
planning
to
use
that
with
delta
rs
the
nice
thing
about
statewide
is
it
also
lets
you
specify
the
spec
with
your
within
your
implementation,
so
it
makes
sure
that
it
makes
sure
that
the
spec
is
always
up
to
date
with
the
implementation,
and
I
think
that
that's
really
that's
something.
That's
it's
not
possible
to
do
in
tla
plus,
so
I'm
pretty
excited
to
learn
more
about
the
project
and
see
if
we
can
adopt
it
for
data
rs.
D
C
Yeah,
that's
a
good
question.
Actually
we
don't
have
such
check.
It's
kind
of
we
are,
but
every
time
when
we
try
to
make
changes
to
this
like
transaction
conflict
detection,
we
need
to
go
through
many
senior
engineers
to
review
what
you
would
like
to
change,
because
it
is
the
core
part
of
the
correctness
of
the
table.
D
C
Soon,
actually,
finally,
I
kind
of
working
on
like
delta,
sharing
server
in
databricks
yeah.
We
have
like
open
source
server,
but
in
order
to
like
make
this
working
and
database,
there
are
a
lot
of
like
work.
I
need
to
do
yeah
because
we
have
like
a
different
like
environment.
You
know
like
prototype,
so
we
need
to
it's
kind
of
probably
I
we
need.
I
don't
really
write
a
lot
of
code.
You
know
infrastructure.
B
B
I've
gotta
put
my
money
where
my
mouth
is
on.
Those
writing.
Lambdas
is
so
fun
and
rest.
I
don't
know
they're
so
simple,
all
right,
I
should
I
should
go
ahead
and
wrap
this
up
so
that
we
can
all
get
back
to
our
respective
day
jobs
or
hacking,
or
what
have
you
ryan?
Thank
you.
So
much
for
your
time.
Qp
denny.
Also
thank
you
so
much
for
your
time.
B
If
you've
got
again,
if
you've
got
further
questions
about
delta
lake
or
where
the
project
is
or
where
it's
going
definitely
join
us
in
our
slack
workspace
or
the
delta
users.
Google
group,
both
of
those
are
linked
from
delta.io
and
yeah.
I
guess
come
join
us
delta,
lake's,
a
fun
project
to
work
on,
and
thank
you
all
for
joining
me
I'll,
see
you
all
later.