►
From YouTube: Delta Lake Community Office Hours (2022-06-09)
Description
Join us on June 9, 2022 at 9:00 AM PDT for the Delta Lake Community Office Hours! Ask your Delta Lake questions live and join our guest speakers, Dominique Brezinski, Tathagata Das, and Gerhard Brueckl alongside Vini Jaiswal from Delta Lake!
Ask us your #DeltaLake questions. These sessions allow our community to ask questions about Delta Lake OSS and get to learn what we are building, planning to build and know about recently released features.
Quick links:
https://delta.io/
https://github.com/badal-io/datastream-deltalake-connector
https://groups.google.com/g/delta-users
A
Youtube
channels-
and
please
have
your
questions
coming,
so
we
can
get
to
it
quickly
for
those
who
are
new
to
the
session.
These
sessions
are
live
and
occur
every
two
weeks
on
thursdays
at
9,
00
a.m,
pacific
or
12
p.m.
Eastern
time-
and
we
try
to
bring
awesome
people
awesome
contributors
to
the
panel
so
that
they
can
answer
your
questions
live.
You
know
for
all
the
delta
questions
you
have,
so
you
can
ask
questions
about
data
like
open
source
software,
how
to
use
it.
A
B
Yeah,
thank
you
dominique
brzezinski,
I'm
a
distinguished
engineer
at
apple.
I
work
in
the
information
security
space,
but
predominantly
on
our
big
data
pipelines
for
security,
telemetry,
for
doing
detection
response
and
been
a
data
bricks
customer
for
four
plus
years
now.
I
think-
and
our
use
case
was
sort
of
the
genesis
for
delta
lake
and
then
things
like
auto
loader
and
a
few
other
kind
of
core
features
in
data
bricks.
B
So
have
had
the
pleasure
of
being
able
to
work
very
closely
with
the
team
over
time
to
refine
and
extend
that
stuff
and
only
recently
got
to
a
point
where
I
could
actually
start
to
do
real
contributions
to
the
open
source
project
as
well.
You
know
given
normal
corporate
politics
and
intellectual
property
stuff,
but
you
go.
A
C
C
His
requirement
of
the
scale
of
handling
data
in
a
transactionally
correct
way,
with
high
quality
at
petabyte
scale
is
what
led
to
building
delta,
and
so
he
he
is
the
the
fundamental
idea
where,
behind
at
this
entire
thing,
that
has
that
started
this
now
four
year
old
journey,
four
plus
year
old
journey.
So
but
along
the
way,
I
have
had
the
privilege
to
build
some
of
the
stuff
in
delta
and
right
now
I
continue
to
work
specifically
focusing
on
the
delta,
open
source
side
of
things
features
connectors
and
stuff.
C
A
D
D
Well,
my
focus
is
mainly
in
the
microsoft
area
and
working
since
this
data
bricks,
since
it
actually
was
available
in
the
azure
cloud,
same
accounts
for
for
delta
lake
and
I'm
contributing
the
power
bi
connected
to
the
delta
lake.
So,
basically
allowing
you
to
read
a
delta
lake
table
natively
in
power
bi
without
the
need
of
having
a
databricks
cluster
running.
C
D
A
D
Yeah,
let
me
start-
although
I
guess
most
of
my
inspiration,
let's
say,
comes
from
from
customer
demands
that
I
most
of
the
time
get
directly
from
them,
and
then
I
say:
okay
well,
that
makes
sense
to
add
to
to
the
connector.
Sometimes
it's
also
like.
If
there
is
some
new
features
that
get
into
delta,
that
I
also
try
to
add
them
as
soon
as
possible.
D
C
As
well,
customer
demand
like
not
just
customers,
just
user
community
demand
like
customers,
is
only
a
small
part.
Database
is
only
a
small
part
of
the
much
larger
community.
So
it's
a
community
many
community
demand
led
to
all
the
new
features
we've
been
adding
in
the
last
few
releases
and
the
next
release.
That's
going
to
happen
so
yeah,
that's
simple!.
A
D
A
C
Yes,
so
I
think
one
of
the
main
connectors
that
is
our
currently
focus
is
basically
flink.
So
about
a
couple
of
months
ago,
we
released
the
first
version
of
flink
sync
to
write
to
delta
tables.
Now
we
are
very
close
to
code
complete,
and
hopefully
we
gonna
release
something
before
the
data
plus
ai
summit
for
the
flink
source,
so
reading
from
delta
tables.
So
with
that,
we
have
the
end
to
end
flink
reading
and
writing
from
delta
tables
complete.
A
From
your
perspective,
right
in
1.2,
we
added
a
lot
of
features
on
optimize
as
well.
As
you
know,
column
mapping
generated
column
support
for
merge.
All
of
these
features.
How?
How
are
what
is
your
take
on
you
know,
implementing
implementing
those
features
for
delta
data
engineering
pipelines
would
love
to
understand
a
little
bit
more
on
how
you
are
either
using
these
features
or
maybe
think
about
how
this
is
revolutionizing.
B
For
us
all,
the
other
stuff
is
super
quality
of
life
being
able
to
you
know
kind
of
remap
columns
and
column
names
having
these
types
of
features
when
you're
having
to
do
data
engineering
across
hundreds
or
thousands
of
input,
data
sets
right
and
and
all
the
schema
complexity
that
comes
with
that
stuff.
Any
features
to
be
able
to
kind
of
make
things
better
or
correct
things.
I
mean
no
matter
how
honest
you
try
to
be.
B
B
B
A
Laid
out
perfectly
done,
these
are
the
features
which
you
know.
People
have
been
getting
excited
and
actually,
as
a
data
engineer
myself
back
then
working
with
a
lot
of
like
time
stamps
and
you
know
having
generated
columns
automatically
generated
for
data
engineers.
I
think
that's
a
that's
a
huge
thing
as
well,
so
just
adding
two
cents.
A
A
C
C
C
Historically,
this
was
another
one
of
those
features
that
was
built
specifically
to
handle
the
scale
that
dom
needed
to
handle
like
searching
like
having
queries,
run
on
terabyte
petabyte
scale,
table
single
table
having
terabyte
plus
of
data
and
stills
returning
within
seconds
to
find
that
few
columns
that
satisfy
that
filter
across
multiple
columns.
It
was
built
specifically
for
dom
and
now.
Finally,
after
this
many
years,
we
are
seeing
the
transition
that
we
are
finally
putting
in
open
source.
It
has
been
merged.
The
next
three
is:
will
have
the
z
order,
this
thing.
C
So
what,
from
the
user
point
of
view,
you
will
be
able
to
run
this
command
called
optimize
the
order
and
that
will
rearrange
your
data
in
the
table
based
on
the
columns
you
want.
It
will
cluster
the
data
based
on
those
columns
and,
after
that,
all
the
filter.
Queries
on
that
column
would
be
much
much
much
faster
because
you'll
be
able
to
eliminate
files
completely
without
having
to
scan
them.
For
that
particular
value,
you
want
in
the
filter,
yeah.
B
Yeah,
it's
so
funny
like
that.
I
think
the
genesis
story-
and
this
this
actually
speaks
tons
for
td's
team
and
michael
and
a
bunch
of
the
other
people
at
data
breaks.
But
when
we
were
originally,
you
know
talking
about
this
kind
of
selective
search
use
case
michael
and
I
were
in
a
room
and
it
was
like.
B
Can
you
run
a
couple
of
like
pieces
of
code
against
this
data
and
I
need
to
generate
some
statistics
and
look
at
some
stuff,
and
so
I
do
this
experiment
and
then
not
long
later,
like
it
comes
back
and
it's
like
here
try
and
run
this,
and
it
was
basically
like
the
alpha
version
of
z
order
right
and
we
run
it
on
a
bunch
of
data,
and
then
we
try
some
searches
by
one
column,
the
other
column.
You
know
values
from
both
columns
and
it
was
outstanding
like
right
away.
B
We
were
able
to
just
exclude
like
90
to
99
of
the
data
within
you
know
very
large
cables,
just
based
on
the
fact
that
you
know
now.
Multiple
columns
were
sorted
in
a
way
that
the
min
max
stats,
you
know,
would
be
sort
of
tight
around
files,
and
it's
likely,
if
we
had
you
know,
values
that
that
were
very
sparse
within
our
data
set,
that
they
would
be
nicely
isolated
into
one
or
a
small
number
of
files
in
the
table.
And
then
you
know
the
delta
reader
would
do
its
thing.
Evaluate
the
metadata.
B
Look
at
the
query
plan
against
the
the
column
stats
there
and
be
able
to
generate
a
physical
plan
right
that
just
targeted
the
parquet
files
that
could
possibly
have
answering
and
being
able
to
do
this
kind
of
z,
ordering
or
clustering
sort
over
that
has
allowed
us
now
to
pick.
You
know
up
to
a
couple
up
to
a
few.
You
know
columns
that
would
be
primary
kind
of
search
columns.
We
use
in
data
sets,
and
then
we
can
z
order
by
that
and
we
effectively
take.
B
B
About
it
is
that
it's
one
of
those
things
that
you
can
choose
to
use
or
not
right,
you
can
choose
to
spend
the
compute
time,
does
the
order
a
table
if
you
get
advantage
out
of
it
or
you
can
not
right.
If
you
don't
do
selective
queries
and
you're,
not
paying
like
a
right
time
penalty
to
always
maintain
an
index
right.
B
If
that's
not
what
you
need,
but
if
you
do,
you
have
the
flexibility
to
kind
of
move
this
in
right
and
and
optimize
your
data
in
this
way,
and
you
also
can
change
it,
which
is
super
important
right.
You
can
take
the
same
table.
You
can
z
order
it
by
you
know
two
columns
and
decide
those
aren't
the
best
columns
and
change
it
to
a
different
two
or
three
columns,
and
that
works
just
fine.
A
B
A
D
I
think
I
just
wanted
to
mention
you
can
also
do
it
on
a
petition
level,
so
you
don't
have
to
do
it
for
the
whole
table.
You
can
just
do
it
for
like
the
most
recent
petitions
that
are
probably
queried
the
most
often
which
at
some
to
some
degree
limits
or
or
decreases
the
impact
on
your
overall
table
and
storage
and
processing
time
that
you
need
to
actually
create
the
indexes
yeah.
C
A
C
So
now
let
me
take
that,
so
let
me
take
it
from
the
protocol.
Point
of
view
is
that
the
delta
protocol
does
not
require
different
logical
partitions
to
be
present
in
different
sub
directories,
but
by
default
the
way
delta
evolved
over
time
and
stuff.
We
d:
we
do
that
to
do
the
actual
subdirectory
symbol
directly
by
default,
to
kind
of
because
we
were
inspired
by
hives
style
of
partitioning,
but
we
don't
have
to
do
that.
B
A
C
Yes,
so
let
me
reinterpret
the
question,
and
hopefully
my
interpretation
is
the
questions
correct.
So
can
you
query
row
level
changes
from
a
delta
table
and
the
quick
answer
is
yes,
and
now
you
can
so
in
the
next
release.
We
another
big
feature:
we're
going
to
release
is
change
data
feed
where
it
is.
You
set
a
table
property
in
delta
table
and
with
that
all
right
operations
that
are
doing
row
level,
modifications
like
merge
or
update,
or
delete
like
this
modifying
a
single
row
or
in
dna
file
and
rewriting
files.
C
Before
this
feature,
it
was
not
very
efficient
to
do
that,
because
you
could
only
read
files
that
were
entirely
rewritten,
because
only
one
of
the
rows
had
changed.
So
you
would
have
to
read
the
entire
file,
which
is
99
unmodified
data
just
to
process
that
single
row
with.
But
with
this
feature,
we
keep
them
separately,
it's
a
so
that
you
can
read
them
efficiently.
It's
opt-in
features.
So
if
you
don't
want
don't
care
about
reading
low-level
changes,
you
don't
have
to
so
it's
completely
opt-in.
There
is
no
cost
associated
with
it.
A
Yeah,
that's
awesome.
Td
cdf
is
the
feature
that
that
is
coming
up
and
we
discussed
last
time.
I
think
we
only
have
three
more
minutes,
we'll
take
one
question
and
we'll
just
give
a
recap
on
what
to
expect
in
three
weeks.
So
there's
a
question
about:
is
it
possible
maintaining
the
indexes
as
online
activity
during
business
hour
or
its
downtime
activity,
so
maintaining
indexes?
C
Can
take
a
crack
at
it,
so
yeah,
so
at
this
point
in
time,
the
indices
that
delta
supports
is
not
it's
more
like
min
max
values
of
each
column
in
each
file.
So
that's
not
a
traditional
index
in
the
academic
sense,
but
but
that
is
very
good
to
eliminate
files.
If
you're
looking
for
a
particular
value,
5
and
a
file
has
only
data
for
that
column,
between
10
and
15,
you
don't
need
to
read
that
file,
so
that
is
how
column
stats.
What
it's
called
is
is
used
now
for
so
business
hours
versus
non-business
hours.
C
It's
it's
all
about
how
you
arrange
the
table
organize
the
data
in
the
table.
If
you
are
like
separating
out
business
and
non-business
hours
into
separate
partitions
like
partition
by
hour
or
something
then
you
can
have,
you
can
potentially
have
different
data
organization
like
one
thing
just
compacted
the
other
things,
the
order,
the
other
things
got
to
be
a
different
thing.
You
could
do
that
to
take
best
advantage
of
the
kind
of
queries
that
you
get
on
business
hour,
data
versus
non-business
hour
data.
A
D
C
A
Yeah,
so
that's
super
important
thanks
for
calling
that
out
td.
We
will
be
releasing
that
through
our
google
group
and
slack
channel
I'll
point.
The
links
in
in
this
youtube
recording
and
please
do
take
a
crack
at
it
if
you
are
active
actively
building
on
desalet,
so
that
would
be
really
appreciated
and
just
as
td
called
out,
we
have
data
and
ai
summit
coming
up
last
week
of
this
month,
and
you
know
on
june
29th
you
will
get
to
see
all
these
faces,
plus
others
in
the
community.
So
please
come
join
us.