►
Description
In this tech talk, we'll explore how Delta Sharing has been extended to support sharing Delta Tables on both Azure and Google Cloud Platform. We'll dive into some of the great enhancements to the Delta Sharing project over the past two releases, including some cool features like query limits, to reduce the size of the dataset you’d like to explore, adding a Share expiration time, automatic refresh of presigned file URLs for long running queries, as well as other enhancements to the Sharing Server protocol.
To conclude, we’ll talk about some of the upcoming milestones in the Delta Sharing project and end with a live Q&A session to get your feedback and to answer some of your burning questions.
*This meetup will be recorded and will be available online here: https://youtube.com/playlist?list=PLTPXxbhUt-YVPwG3OWNQ-1bJI_s_YRvqP
A few resources:
https://delta.io/sharing/
https://github.com/delta-io/delta-sharing
Speakers:
Will Girten is a Sr. SSA at Databricks, where he's helped some of the largest customers at Databricks build modern, enterprise Delta Lakes in the cloud. He specializes in building efficient and reliable ETL pipelines for fast data engineering and BI workloads. Prior to Databricks, Will worked as a Data Architect helping federal customers build intelligent data lakes in HealthCare and Government verticals.
Ryan Zhu is a Staff Software Engineer at Databricks, an Apache Spark committer and a member of the PMC. He is one of the core developers of Delta Lake, Delta Sharing and Structured Streaming.
A
B
Okay,
excellent,
so
thank
you
all
so
much
for
joining
today's
webinar
today.
I'd
like
to
talk
about
some
really
cool
features
that
have
been
added
to
the
delta
sharing
project
over
the
past
couple
of
releases,
most
notably
delta
sharing,
has
been
extended
to
support
sharing
delta
tables
on
azure,
blob
storage,
adls,
gen
2,
as
well
as
google
cloud
storage.
B
B
Sure,
thanks
ryan
and
just
brief
introduction
about
myself.
My
name
is:
will
gurdon
I'm
a
senior
ssa
databricks
based
out
of
new
york?
I
earned
my
computer
engineering
degree
at
university
of
delaware
go
blue
hands.
I
joined
databricks
back
in
2019
and
since
then,
I've
helped
some
of
the
largest
customers
at
databricks
build
modern
enterprise
delta
lakes.
In
the
cloud
my
background
is
mainly
data,
engineering
and
business
intelligence
and
then,
in
a
past
life
prior
to
databricks.
B
B
So
also
you
know
kind
of
what
are
the
motivations
behind
using
delta
sharing?
Well,
there's
a
couple
problems
worth
solving
one:
is
you
know
being
able
to
share
your
data
across
your
organization
or
maybe
across
different
companies
or
agencies?
If
you
work
in
the
in
the
government
vertical
so
a
lot
of
the
times
when
I
work
with
customers,
I
see
two
common
patterns
across
customers
when
they
try
and
share
the
data
across
the
organization.
B
So
this
is
all
well
and
fine
as
long
as
you
and
others
continue
to
purchase
the
data
warehouse
vendor.
So
it's
not
very
good
as
far
as
vendor
lock-in,
but
perhaps
you
maybe
you
have
maybe
a
budget
constraints
or
maybe
a
long
procurement
cycle,
or
maybe
data
or
there's
a
different
data.
Warehouse
fender
that
promises
more
efficient,
faster
speeds.
B
So
the
the
real
solution
here
is:
don't
copy
the
data
at
all,
so
that
really
is
kind
of
the
main
motivation
behind
delta
sharing
and
from
you
know,
some
of
the
main
goals
that
we've
created
for
the
delta
sharing
project
is
is
one
is
that
we
wanted
to
be
able
to
to
share
the
existing
live
data
as
it
lands
in
your
enterprise
data
lake
no
need
to
copy
it
out.
Etl
entered
into
a
vendor.
B
B
So
the
delta
sharing
protocol
has
strong
security,
modern
and
governance
built
in
as
well
and
then.
Last
but
not
least,
we
wanted
to
be
able
to
ship
to
scale
out
to
massive
data
sets
and
we'll
see
in
just
a
second
how
we
achieve
that.
But
basically
there
are
a
lot
of
use
cases
in
all
sorts
of
different
verticals,
where
the
data
sets
being
shared
or
in
the
terabytes
or
even
petabytes
of
data,
so
these
were
kind
of
the
main
primary
goals
behind
delta
sharing.
B
B
So
it
all
starts
with
the
delta
lake
being
shared
by
a
provider
and
then
in
front
of
that
is
a
delta
sharing
server
that
kind
of
acts
as
the
gatekeeper
it
determines,
whether
or
not
that
data
recipient
on
the
right
has
access
to
a
particular
data
data
set
or
subset
of
the
data,
and
it
enforces
access
to
to
the
actual
data
and
then
on.
The
far
right
is
is
the
data
recipient
and
they
could
use
any
any
sharing
client
that
implements
the
delta
sharing
protocol.
B
So
that's
really
kind
of
the
the
major
elements
of
delta
sharing
protocol
and
how
it
actually
works
on
the
hood
is
say
that
a
data
recipient
wants
to
go
and
access
or
read
some
of
the
some
of
the
data
that's
being
shared
with
them.
So
they'll
begin
by
sending
a
submit
sending
a
request
to
the
delta
sharing
server.
In
this
example,
we
have
a
data
provider,
say
it's
a
retail
provider.
That's
sharing.
Real-Time
information
about
purchase.
B
B
Whether
or
not
this
this
recipient
is
authorized
to
access
the
data,
and
in
this
case
we
can
imagine
that
they
do
have
access
and
then
from
there
the
delta
sharing
server
will
go
out
and
it
will
read
the
different
files
that
have
been
persisted
to
cloud
storage,
and
this
doesn't
necessarily
have
to
be
all
of
the
cloud
all
of
the
file
objects.
It
could
be
a
subset
of
the
data
as
well,
so
that
they
only
want
to
query
a
particular
part,
partition
or
date
range
and
then
from
it.
B
The
delta
sharing
server
will
generate
short-lived
pre-signed
file,
urls
and
return
that
back
to
the
data
recipient
and
in
the
past
two
releases.
We've
actually
extended
this
to
other
cloud
vendors
as
well,
in
particular,
azure,
blob,
storage,
adls,
gen
2,
as
well
as
in
the
latest,
release
google
cloud
storage
as
well.
B
It's
really
completely
abstracted
from
them,
and
then
you
know
kind
of
on
that
fourth
objective
of
delta
sharing
is
we
wanted
to
be
able
to
provide
a
protocol
for
sharing
the
data
that
would
scale
out
to
terabytes
of
data
and
the
way
that
we
do
that
is.
We
can
take
advantage
of
the
cloud
vendors
themselves
to
be
able
to
cost
effectively
and
efficiently
share
the
data
and
parallelize
that
read
so
instead
of
the
data
traversing
through
the
the
delta
sharing,
those
pre-signed
urls
can
be
read
in
parallel,
using
something
like
pi
spark.
B
And
then
also
just
kind
of
talk
about
the
future
road
map.
Some
of
the
upcoming
features
that
will
be
added
to
the
delta
sharing
project
is
a
common
use.
Case
is,
maybe
you
want
to
see
all
of
the
different
changes
that
have
happened
to
a
shared
data
set,
so
in
the
future
we
will
also
support
being
able
to
read
the
incremental
changes
from
a
shared
data
set
as
well.
B
Okay.
So
that's
enough
talking
that
kind
of
covers.
You
know
what
are
the
motivation
behind
delta
sharing
and
how
it
works
on
the
hood,
I
wanted
to
get
into
an
actual
concrete
example
of
using
the
delta
sharing
project
and
kind
of
where
to
two
different
play.
Diff
play
two
different
roles
here
as
the
data
provider
and
did
a
recipient.
B
I
work
as
a
part
of
a
data
science
team
and
I've
created
my
data
set
stored
it
out
into
multiple
clouds
here,
and
I
like
to
provide
this
or
share
this
data
set
to
multiple
recipients,
whether
it
be
you
know,
a
different
team
with
my
organization,
or
maybe
it's
a
different
outside
agency
or
a
different
company,
so
it
all
begins
with
that
delta
sharing
server.
So
in
this
example,
it
starts
with
our
delta
sharing
server
configuration
I've
basically
like
to
share
two
tables
here.
B
One
is
a
sampled
cova
data
set
that
has
all
the
cases
that
have
happened
over
time,
as
well
as
a
second
data
set.
That
has
the
cases
with
the
agent
included
over
time
as
well.
So
one
is
stored
out
in
adls,
gen
2
and
then
the
other
is
actually
stored
out.
Google
cloud
storage,
so
in
this
example,
I've
created
a
share,
a
delta
share,
called
webinar
share
and
under
that
I've
created
a
schema,
called
sample
copa
data
and
I'm
sharing
both
of
these
tables
and
they're.
They
both
both
the
data,
exist
on
two
different
clouds.
B
B
I've
created
a
delta
sharing
server,
that's
running
locally
on
my
laptop,
which
is
just
fine
for
demo
purposes,
so
that
takes
care
of
providing
the
data
to
recipients.
So
I'm
going
to
theoretically
change
hats
here
and
pretend
I'm
a
data
recipient
as
well,
so
say
that
I'm
in
a
different
team
on
the
analytics
team
within
my
company.
B
I
know
that
a
different
team
has
shared
access
to
both
these
data
sets.
It
all
begins
with
basically
my
sharing
profile
and
it's
very
simple:
it's
just
a
quick
short
configuration
file
that
includes
the
version
of
my
credentials,
the
endpoint
or
location
of
my
delta
sharing
server
then.
Lastly,
my
personal
access
token
for
accessing
that
data,
and
that's
it
so
now
I
have
basically
the
information
for
connecting
to
my
delta
sharing
server.
I
can
use
whatever
syntax
is
most
comfortable
to
me.
D
B
B
B
I
then
create
a
sharing
client
that
allows
me
to
access
the
data
and
manipulate
it
and
then
from
there.
I
need
to
point
to
which
table
within
that
delta
share.
I
would
like
to
work
with
so
this
example.
I'm
going
to
I'm
going
to
load
in
the
data
set
called
time,
which
is
covered
cases
over
time,
I'm
going
to
load
it
into
a
pandas
data
frame
because
I,
like
pandas
api,
it's
most,
I'm
most
comfortable
with
it
and
then
from
there.
B
I
can
basically,
you
know
basically
read
that
those
cloud
files
that
are
stored
out
in
adls
gen
2..
I
don't
really
care
about
the
implementation
details.
In
fact,
I
didn't
even
know
that
it's
stored
out
in
adoles
gen
2.
I
just
know
that
there's
a
data
set
called
time-
that's
been
shared
with
me,
so
this
example,
you
know
just
a
few
seconds.
It's
reached
out
to
adolescent2
brought
down
the
data
and
loaded
into
memory
and
I've
displayed.
B
You
know
just
a
subset
of
this
and
then
similarly
say
that
I
want
to
also
work
with
that
that
table
that's
stored
out
in
google
cloud
storage.
This
should
look
very
similar
to
the
other
one.
The
only
thing
that's
really
changed
is
the
name
of
the
table
so
again
I'll
go
ahead
and
run
this
read
it
in
as
a
painless
data
frame
and
you'll
see
that
this
goes
out,
accesses
the
data
from
google
cloud
storage
and
loads
it
into
memory
identical
source
code.
B
The
only
thing
that's
changed
is
is
the
name
of
the
table
and
all
the
implementation
details
about
where
the
data
is
actually
stored,
how
to
fetch
it.
It's
all
hidden
from
it
makes
it
very,
very
simple:
to
read
your
data
share
it
and
not
have
to
have
any
friction
about
standing
up
your
own
own
server.
B
You
know
sizing
it
out
really
dealing
with
access
permissions
things
like
that,
and
then
I
can
also
do
a
little
bit
more
intricate
things
too,
as
well
so
say
that
I
want
to
basically
take
these
two
data
sets
and
join
them
together.
Maybe
maybe
produce
some
sort
of
visualization
or
be
I
report
it
into
this.
I
can
also
do
that
too.
Just
use,
what's
comfortable
with
me
in
this
case
pandas
api
to
basically
merge
these
two
data
sets
together
and
then
also
one
other
thing
to
kind
of
call
out.
B
That's
been
released
in
the
past
two
releases.
Is
this
optional
parameter
here
in
your
api
for
for
the
pandas
api
and
then
also
the
pi
spark
connector?
Is
the
ability
to
load
in
maybe
just
a
subset
of
your
data
set?
Maybe
your
data
set's
huge,
and
you
only
want
to
you-
know,
take
a
peek
at
your
data
set.
You
can
you
can
pass
in
this
optional
parameter
here
called
limit
limit
this.
B
I
basically
loaded
that
same
those
two
tables
again
and
I'm
using
the
the
spark
connector
here
in
the
delta
sharing
project,
and
I
can
drop
in
my
sequel
that
I'm
most
comfortable
with
here.
I
I've
just
used
spark
sql
here
I
can
drop
in
my
anti-sequel,
I'm
most
comfortable
with
and
then
last
but
not
least,
one
of
the
really
cool
improvements
that
we've
added
in
the
past
couple
releases
is
the
ability
to
to
quickly
check
whether
or
not
you
have
you
still
have
access
to
a
share.
B
If
you
recall
an
architecture
diagram
that
delta
sharing
server
will
generate
short-lived
urls,
so
perhaps
you
know,
maybe
a
couple
of
days
have
it
elapsed,
and
maybe
you
need
to
check
whether
or
not
you
still
access
to
that
shared
data
set.
Well,
we've
made
it
very
simple
to
to
to
check
with
that.
You
still
have
access
to
a
share.
You
can
simply
go
out
to
your
delta
sharing
server
and
then
consume
this
api
endpoint,
which
is
basically
your
your
delta,
sharing
server
url
shares
and
then
the
the
name
of
your
share.
B
So
I'm
just
sending
a
quick
get
request
here
to
check
whether
or
not
I
still
have
access
to
this
data
set,
because
I
just
loaded
this.
I
just
started
this
sharing
server
a
moment
ago.
I
still
have
access,
but,
as
you
can
see,
if
I
just
change
this,
the
share
name
to
something:
that's
invalid
and
I
rerun
this
you'll,
also
notice
that
we've
also
improved
the
the
error
messaging
as
well
within
the
delta
sharing
protocol
in
the
past
couple
of
releases.
B
B
So
that's
really,
you
know
at
a
high
level.
You
know
how
the
delta
sharing
server
works
for
the
hood.
You
know
what
are
the
main
motivations
behind
the
delta
stream
project
and
just
kind
of
walked
you
through
a
couple
of
the
the
cool
features
that
have
been
added
over
the
past
couple
of
releases,
so
just
wanted
to
pause
here
to
see
if
we
have
any
questions
on
the
line.
D
D
I
was
actually
able
to
answer
right
away,
but
some
of
them
would
so
I
figured
this
if
we
got
time
for
questions,
let's
go
ahead
and
do
that
right
now,
so,
first
from
linkedin
there's
a
great
question
from
rohan
is:
where
is
this
python
hosted,
like
you
know
the
code
that
you're
playing
said
and
he's
interested
in
under
knowing
more
about
how
to
host
the
server
and
can
we
do
insert
and
update
data
as
well?
So,
let's,
let's
break
this
down.
D
B
D
Perfect,
okay
and
then
they're
interested
in
knowing
how
to
host
the
server.
So
I
was
about
to
put
in
the
the
sharing
of
the
the
instructions
on
how
to
start
the
server
yourself.
So
I'm
gonna
do
that
right
now,
but
and
can
do
we
support,
insert
update
of
data
as
well,
and
I
we
know
the
answer,
but
I
figured
I
have
one.
You
guys
answer
it
so.
C
Yeah,
I
can
answer
this
so
basically,
with
the
data
sharing
actually
involved,
two
parts,
one
is
the
data
provider,
the
other
ones
like
the
data
recipient
for
data
recipient
countries,
with
only
you
can
now
modify
the
table.
Let's
share
to
you,
but
for
data
provider
based
on
the
table,
they
can
do
whatever
operations
on
the
data
table
using
like,
for
example,
storage
credentials
and
then
so,
but
for
data
recipient
part.
This
is
just
good
on
me.
D
Perfect
perfect
and
then
actually
related
that
there's
a
question
that
came
through
on
zoom.
I
believe,
let
me
go
ahead
and
find
it
here.
Sorry
about!
Oh
sorry,
here
we
go.
What
does
what
type
of
authentication
does
the
sharing
server
currently
support?
So
we've
talked
about
the
pre-signed
urls
from
the
standpoint
of
the
of
getting
access
to
the
data
itself,
but
then,
in
terms
of
how
does
the
authentication
work
specifically
with
the
open
source
sharing
server?
So
could
we
provide
a
little
context
on
that?
Please.
D
Perfect,
perfect,
perfect.
We've
got
a
lot
of
other
questions,
but
I
know
that
there's
still
plenty
of
stuff
that
you
guys
want
to
talk
about.
So
why
don't
we
switch
back
to
the
presentation
I'll
do
my
best
to
answer
the
ones
through
the
the
various
channels
and
then,
if
we
still
have
time
later,
I'll
bring
them
back
up
for
us
to
answer
live
okay.
B
Yeah
sure,
just
real
quickly
about
that
last
question.
I
think
we
missed
one
about
how
do
we
install
or
or
deploy
a
delta
sharing
server?
So
we've
made
it
really
really
simple:
you
can
actually
deploy
this
as
a
docker
container.
You
can
run
it
in
anywhere
and
in
fact
you
don't
need
to
build
a
docker
container
yourself
if
you
just
go
out
to
the
readme
there's
instructions
on
how
to
download
a
hosted
copy
of
the
docker
container
from
docker
registry,
and
you
can
basically
use
that
to
send
up
your
own
dockers.
B
A
D
I'm
sorry,
I
thought
there
was
a
there's
another
section
you
were
gonna
show
my
my
apologies,
okay.
Well
then,
let's
continue
on
with
the
questions.
My
apologies
for
missing
that.
Thank
you.
Let's
see
here
all
right,
so
in
that
case
let
me
pull
some
of
the
older
questions,
because
I
do
want
to
actually
call
that
out.
Okay,
for
example,
there's
a.
B
C
Right
now,
the
open
source
one
doesn't
follow
these
features,
but
actually
you
can,
for
example,
deploy
the
open
source
data
sharing
server
behind
the
proxy,
and
then
you
can
actually
leverage
this
thing
like
an
open
source
process
server
to
support
all
these
features
everywhere.
You
have
at
the
time,
like
a
customer,
actually
is
already
doing
like
a
deploy
data
chain
server
like
this.
Just
behind
the
process
to
to
have
like
offers
like
a
traditional
like
server
capabilities.
D
Yes,
and
one
thing
we
always
are
up
for:
don't
forget,
okay
and
I'll,
we'll
make
sure
to
post
this
as
well.
Join
us
at
the
delta
user,
slack,
okay,
there's
a
delta
sharing
channel
you're,
more
than
welcome
to
join
us
there,
and
also,
if
you
have
some
ideas
about,
for
example
like
when
you
called
out
with
the
rate
limiting
part
yeah,
you
know
we're
always
up
for
prs.
We
love
pull
requests,
so
this
is
an
open
source
project
so
by
all
means.
Okay,
so
just
want
to
call
that
out
all
right.
D
The
next
question
is
that-
and
I'm
actually
I'll
tackle
this
by
myself-
is
that
do
you
plan
to
support
orc,
in
addition
to
part
k,
right
and
right
right
now
and
ryan?
Correct
me
if
I'm
wrong,
but
delta,
sharing
itself,
there's
nothing
actually
inherently
about
delta
sharing
that
prevents
it
from
happening.
It's
just
that
the
vast
majority
of
people
are
asking
for
the
delta
tables,
mainly
enough.
The
name
probably
helps
with
that
and
parquet.
D
So
that's
what
we're
starting
again,
if
there
is
a
lot
of
interest
from
the
community
for
orc
support,
create
an
issue
create
some
prs.
We'd
love
to
have
the
feedback,
and
just
right
now
at
least
there
has
not
that
while
there
certainly
has
been
some
people
asking
for
it
honestly,
it
hasn't
been
that
big.
So
that's
the
reason
why
it's
not
been
a
big
community
push
yet,
but.
C
C
That's
why
we
use
pita
delta
lake
to
in
this
deodorant
protocol,
and
actually
you
can
do
this
like
offline
converting,
for
example,
you
can
convert
a
packet
table
to
a
theoretical
table
easily
using
like
open
source
data
like
projects
and
after
that
the
for
example.
We
don't
need
to
leave
the
files.
We
just
read
the
delta
lag
transaction
law
to
get
all
the
files
in
the
table
and
then
generate
the
present
ui
return
to
the
cons.
D
Yes,
so
to
over
ryan's
point,
basically
we're
talking
about
cloud
cloud
object
stores
because
we're
basically
getting
web-based
apis.
The
listing
of
that
is
going
to
be
erroneously,
painful
right
and
so
the
law,
the
delta
transaction
log
itself,
contains
a
list
of
files.
So
that
way,
anything
that's
trying
to
access
the
delta
table
can
read
knows
exactly
the
path
it's
supposed
to
hit
in
order
to
get
all
that
all
that
stuff.
All
the
files.
D
Excuse
me,
okay,
so
that's
the
reason
why
we're
going
down
that
route,
primarily
when
you're
hitting,
for
example,
our
delta
sharing
server,
that
the
public
one
that
we
share
for
everybody
to
test
it
out
immediately
enough
they're,
not
that
many
files
inside
there,
so
it's
gonna
come
back
relatively
quickly,
whether
you're
talking
about
the
transaction
log
or
not?
Okay,
so
so
daryl,
our
friend
our
friend,
daryl,
also
asked
the
questions
like
hey.
D
Is
there
a
way
to
handling
schema
drift
as
part
of
delta
sharing,
like
you
know,
and
at
least
in
terms
of
at
least
our
attempt
to
answer
is
like
well,
that's
why
we
have
delta
tables
in
the
first
place
right.
The
delta
tables.
Allow
for
both
schema
enforcement
and
schema
evolution,
so
you
can
literally,
as
part
of
the
audit
history
of
a
delta
table.
You'll,
actually
see
all
that
information
inside
there
right
from
the
get-go,
so
daryl.
D
D
The
delta
uses
slack
and
specifically
inside
there,
there's
a
delta
sharing
channel.
Okay,
all
right,
let's
see
so
oh
right,
daryl
did
follow
up
with
one
of
his
questions
that
which
I
thought
was
really
interesting.
Was
that
he's
wondering
if
there
is
the
ability
for
eventually
delta
sharing
to
be
able
to
provide
compute
as
part
of
the
provider
right
now,
there's
no
compute
included
within
delta
sharing
delta
sharing
is
the
file
systems
and
it's
a
service
for
authentication
purposes
right,
but
there's
no
compute.
D
C
I
list
a
great
point
actually,
so
I
think
I'd
suggest
you
to
open
a
issue
on
the
github,
so
we
can
discuss
the
details
regarding,
for
example,
how
to
support
a
list.
For
example,
if,
if
we
been
since
we
discussed
computer,
we
need
to
figure
out
how
to
run
the
computer.
Where
is
the
cluster
we
can
use
to
run?
There
are
a
lot
of
details,
probably
better
to
discuss
in
like,
for
example,
a
github
issue.
D
Yes,
so
exactly
and
they're
also
daryl's,
following
what's
the
q,
a
which
is
all
the
data's
done
by
reading
the
client
today,
we,
we
are
absolutely
in
agreement
with
you
right
now
as
the
early
beginnings
of
the
delta
sharing.
That's
why
we,
we
actually
purposely
did
it
this
way,
they're,
just
like
ryan
called
out.
There
are
multiple
potential
approaches
in
terms
of
actually
us
offering
compute
as
well.
Usually
what
it
comes
down
to
is
that
there's
also
the
factor
of
costs?
D
Do
you
do
this
as
a
docker
vm
that
you
spin
up
on
your
own,
you
know
container,
then
you
have
to
incur
the
cost,
what
what
the
what
the
egress
costs
that
are
associated
with
it.
So
because
of
all
these
different
issues.
This
is
there's
actually
a
lot
of
good,
solid
debates
on
that.
So
because
of
that,
we
definitely
want
to
have
the
conversation
in
an
open
forum,
like
I
said,
like
ryan,
called
out
in
the
github
issue,
so
we
can
discuss
what
these.
D
What
do
we
think
is
the
best
route
right
and
we
can
over
time,
even
produce
a
a
a
proposal
that
we
all
go
work
together
to
get
that
to
figure
out
the
best
approach
to
that
okay,
so
we
actually
still
have
lots
of
other
questions
okay
by
the
way.
So
let
me
go
ahead
and
switch
over
to
linkedin,
because
right
now
I've
been
focusing
mostly
on
zoom.
So
let
me
go
ahead
and
switch
back
to
linkedin.
D
Let's
see
sorry,
I'm
actually
rolling
back,
because
we,
I
skipped
a
lot
of
questions
by
accident.
So
by
the
way,
I'm
gonna
start
with
these
questions.
If
I
missed
your
question
on
linkedin,
please
let
me
know
and
just
type
it
again.
I
apologize.
I
was
not
trying
to
skip
it.
I
just
might
have
missed
it
as
I'm
scrolling.
Okay,
so
srikant
asked
a
good,
solid
question
in
terms
of.
D
B
There
is,
there
is
in
fact
so
yeah
the
the
api,
whether
you
use
python,
for
example,
you
could
list
out
all
the
tables
that
are
within
a
share.
That's
been
shared
to
you
and
you
can
quickly
just
browse
to
see
which
which
data
sets
have
been
shared
to
you,
and
then
you
could
use
those
cool
limit
parameters
to
just
quickly
peek
at
the
data
set
rather
than
loading.
The
full
data
set
in
memory.
D
Cool,
I
think
that
answers
that
perfectly
next
question
is
from
kayati.
I
I
apologize
if
I
said
your
name
incorrectly,
can
you
share
how
access
permissions
work
on
like,
for
example,
like
a
fine
grain
level?
Is
there
access
permissions
to
like
the
partitions
of
the
delta
table
that
are
available
in
delta
sharing
right
now,.
C
Yeah
currently
we
don't
have
fine
grain
access
control,
but
since
this
is
an
open
source
project,
we
would
like
to
hear
your
feedback
and
you
can
create.
You
can
have
an
issue,
and
maybe,
if
you
already
have
some
great
idea,
you
can
also
open
a
full
request
and
then
work.
We
can
work
with
you
and
then
to
make
sure
we
get
this
support
in
like
future
yeah.
D
Yeah,
that
is
a
very
common
request
by
the
way,
so
absolutely
follow
up
and
look
at
the
issues
that
it
currently
exists
and
or
if
they
don't
suffice,
then
absolutely
and
we
invite
you
to
go,
create
a
new
one
and
then
again,
if
you
have,
if
you
want
to
do
pr,
we're
always
up
for
prs,
but
the
context
is
yes
providing
over
time.
Well,
I'm
not
going
to
quote
what
the
road
map
looks
like
right
now,
yes
over
time.
D
That
is
absolutely
something
we
are
we're
going
to
be
working
on
we're
just
again
trying
to
there
have
been
many
other
requests
that
we've
been
tackling
first
at
the
end
of
like
priorities,
namely
the
ones
as
part
of
this.
Today's
webinar
as
part
of
point
three,
the
azure
support
and
point
four,
the
google
cloud
support,
just
because
those
were
viewers,
the
massive
ask
from
the
community
for
those
things.
So
that's
the
one.
D
D
It's
a
it's
a
it's
an
open
source,
github
project
that
actually
implements
the
delta
sharing
protocol.
Using
here
we
go
the
rust
api,
and
so
we
can
potentially
utilize
bindings
with
node.js
and
everything
else
off
of
that.
So
whether
you
want
to
utilize
riverbank
or
you
want
to
do
a
pr
directly
with
delta
sharing
we'd
love
to
have
it
so
not
disagreeing
with
your
rohan
love
the
idea
of
node.js.
Just
simply
saying
it's
not
right.
It's
not!
We
haven't
been
from
a
priority
perspective.
It's
not
it's
not
there!
Yet!
Okay!
So
my
apologies!
D
Let's
see,
I'm
gonna
read
this
other
question.
It's
a
good
question
from
mohitz,
but
I
just
want
to
make
sure
I
got
it
summarized
correctly.
Okay,
you
know
I'll
actually
break
it
down
to
multiple
questions.
Okay,
so
in
production
do
we
need
to
set
up
a
vm
that
will
act
as
a
delta
sharing
server
and
so
the
actually
I'll
I'll
put
it
this
way
in
development.
You
will
too
okay.
D
So
if
you
want
to
set
up
your
own
sharing
microsoft,
that
is
correct,
you
will
want
to
set
up
a
vm
of
some
type
that
basically
goes
ahead
and
acts
as
your
delta
sharing
server
the
I
already
popped
in
the
instructions
on
how
to
do
it,
but
just
like
we'll
have
called
out
there
also
is
a
docker
container
that
you
can
utilize
as
well.
Okay,
so
now
I'm
going
to
be
a
very.
D
B
Yeah,
a
number
of
reasons
you
know
take
advantage,
taking
advantage
of
that
delta
format
so
being
able
to
take
advantage
of
the
data
skipping
aspects
as
well
as
like
ryan,
talked
about.
If
you
have
a
large
data
set
where
you
need
to
regularly
list
out
your
objects,
that
is
very
inefficient,
and
then
you
also
pay
as
a
consumer
for
those
api
calls
as
well.
So
lots
of
benefits
to
using
delta
sharing
there.
C
Yeah,
I
think
another
pointer
is
basically
if
you
share,
like
your
storage
directory
to
another
user.
Actually,
I
need
to
use
the
specific
like
doing
to
clients
to
read
this
means
you
need
to
let
your
customer
or
your
user
need
to
look
for
the
support
in
different
languages,
for
example,
if
they
use
go,
I
don't
know
whether
it's
able
they
are
able
to
use,
go
to
access
or
not
right
now,
but
why
not?
Data
studio
is
like
a
purely
like
http,
which
most
of
language
will
support.
C
Http
tdp,
like
a
client
and,
for
example,
you
can
use
http
client
to
talk
to
data
streaming
server.
You
will
get
a
lot
of
like
presenter
uis,
which
are
also
in
http
format,
and
then
you
just
use
the
same
magic,
tv
kind
of
to
read
this
data,
and
so
all
the
languages
can
support
the
data
sharing
easily.
D
Right
and
just
to
over
emphasize
ryan's
point
exactly
because
we're
talking
about
web-based
http
apis.
That
is,
that
question
about
node.js,
for
example,
right
sure
we
haven't
built
ourselves,
but
it
is
using
an
http
api.
So
anybody
who
wants
to
build
build
a
pull
request
for
node.js
to
talk
to
it
is
relatively
straightforward
right,
and
so
that's
the
context
right.
It
gives
you
a
lot
of
flexibility
that,
even
though
you
are
setting
things
up
in
abs,
gen
2,
your
client,
your
customers,
don't
require
any
access
to
azure
whatsoever.
D
It's
not
a
requirement
at
all,
and
even
for
those
that
do
require
azure.
Perhaps
you
want
a
different
mechanism
to
access
that
data
that
doesn't
require
specifically
using
adl's
gen
2
shares
right,
for
example,
using
a
web-based
protocol.
Well,
then,
again,
this
is
the
advantage
of
such
approach
right,
so
it
gives
you
that
right
balance
of
extensive
flexibility,
yet
still
leveraging
your
scalable
cloud
storage.
C
Yeah
and
also
in
addition,
it's
also
about
like
a
multi-cloud
support,
for
example,
in
the
future,
you
may
be
move
your
data
from
adl
gentle
to
like
a
google
cloud
storage
and
then
actually
the
client
doesn't
need
to
change
anything
because
everything
are
the
same.
They
all
use
the
same.
Http
click
to
talk.
D
Perfect,
okay,
so
vikash
has
actually
asked
a
question.
That's
a
little
close
to
my
heart,
which
is,
and
I'm
assuming
I
have
a
use
case-
to
consume
a
delta
table
data
with
an
analysis,
services
model
and
an
aes
model.
Okay,
so
for
some
of
you
may
or
may
not
know,
I
actually
used
to
work
with
the
analysis
services
team
in
the
announcement
services
team.
D
Excuse
me,
okay,
so
is
this
doable
delta
sharing
now
not
exactly
yet,
because
aas
by
itself
specifically
doesn't
do
that,
but
there
is
a
power
bi
connector
that
connects
directly
to
delta
sharing.
Now
that's
been
in
beta.
Since
I
want
to
say
two
months
ago
I
think
now,
but
maybe
I'm
a
little
hazy,
but
so
my
apologies
on
the
timing,
but
nevertheless
there
is
in
beta
a
power
bi
connector
directly
to
delta
sharing.
D
So
you
could
definitely
utilize
power
bi
to
access
that
and
perhaps
use
that
as
your
ability
to
refresh
the
model
if
you're
working
with
the
ais
team
and
hi
everybody
over
there.
If
you
want
to
go
ahead
and
work
with
us
directly
to
have
af
talk
directly,
not
even
using
power
bi
to
do
it
glad
to
work
with
you,
the
protocol
is
straightforward.
D
D
Okay,
so
hopefully
that
helps
answers
your
question
because,
and
yes,
my
name
is
danny
lee
you're,
more
than
welcome
to
ping
me
directly
on
the
delta
slack,
just
because
you're
hitting
close
close,
my
heart
on
this
one,
okay,
we're
gonna,
probably
time
for
just
a
few
more
questions
before
karen
tells
me
that
we
have
to
stop
talking.
So
let
me
at
least
try
to
talk
about
those
ones.
Can
we
handle
data
type
changes
in
addition
of
columns
when
writing
a
delta
table?
Now
this
is
not
specific
to
delta
sharing.
D
Now,
by
the
way,
this
is
actually
delta
lake
itself
and
so
delta
lake
itself
allows
for
both
scheme
enforcement,
which
basically
prevents
you
to
do
any
of
these
data
type
changes,
because
you
don't
want
to
make
those
changes
and
also
allows
you
to
go
ahead
and
actually
allows
you
to
evolve
your
schema.
So
you
can
make
those
changes
so,
when
you're
working
with
delta,
sharing
within
the
context
of
a
delta.
B
D
All
right,
so
you
know
what
I
think:
that's
it
for
questions
I
apologize,
but
I
figured
there's
a.
I
know,
there's
a
lot
more
so
again,
I'm
going
to
answer
my
usual
question:
join
us
at
the
delta
user,
slack
delta,
sharing
channel.
We
we,
the
three
of
us
and
for
that
matter,
others
are
on
there
pretty
regularly
and
we'll
answer
your
questions.
So
karen,
why
don't
you
close
this
up
before
I
go
ahead
and
take
this
completely
off
course.
A
Thanks
donnie,
no,
that's
great!
I
it's
always
awesome
to
see
so
many
questions
coming
in
and
yeah
we'd
love.
I
think
the
moral
of
the
stories
we'd
love
to
have
you
join
the
delta
lake
slack
and
everyone's
excited
there
to
engage
with
you
and
answer
your
questions.
If
we
weren't
able
to
get
to
them,
I
dropped
the
link
to
the
the
delta,
like
slack,
so
you
have
that
there
and
then
also
two
for
the
recording
is
available
right
after
this
session.