►
From YouTube: Simon + Denny Ask Us Anything - December 13, 2022
Description
Join us for the monthly series "Simon and Denny - Ask Us Anything!" where we will answer your data engineering questions from building a data platform to ingestion to ETL to analytics. With our background in SQL Server and BI to Apache Spark and Delta Lake - we want to show you how to build your own lakehouse.
As this session is interactive, come prepared to ask questions all throughout the session! Be prepared for another geeky, trans-Atlantic event from two data nerds.
Learn more about Delta Lake: https://delta.io/
Join us on Slack: https://go.delta.io/slack
Denny Lee: https://www.linkedin.com/in/dennyglee/
Simon Whiteley: https://www.linkedin.com/in/simon-whiteley-uk/
Delta Lake Releases: https://github.com/delta-io/delta/releases
A
B
A
A
B
A
B
B
A
Real
good,
we
are
good
sorry,
I
was
I
end
up
clicking
the
wrong
button
on
Zoom,
so
I
end
up
doing
something
else
completely
that
had
nothing
to
do
with
anything.
So
my
apologies,
so
all
right,
I
think
we're
ready
to
start
the
show.
So
we
are
on
live
on
LinkedIn
on
zoom
and
on
YouTube,
just
to
make
sure
things
are
as
confusing
as
possible.
So
for
starters,
if
you
have
any
questions,
please
go
ahead
and
post
them
directly
into
the
the
chat
in
either
LinkedIn
or
YouTube
or
Zoom,
so
yeah.
A
That's
it
really
and
because
remember
this
what's
today's
session,
this
is
the
session
of
well
Simon.
Denny.
Ask
us
anything!
So
you
know
you
can
literally
go
ahead
and
ask
us
anything.
So
literally,
ask
your
questions
here.
Let
me
check
to
make
sure
that
LinkedIn
actually
works
here
today,
because
we've
been
noticing
some
comment
issues.
A
But
right
now
we're
a
little
less
into
those
controversial
topics.
I
guess
and
it's
more
a
matter
of
like
hey:
do
you
have
any
questions
relating
to
I?
Don't
know
type
2,
std2,
Data
warehousing
techniques,
because
after
all,
Simon
and
I
are
both
old
SQL
Server
hounds
pounds
is
that
okay
I
think
it
is,
and
then
I
did
want
to
do
some
quick
announcements
while
we're
waiting,
I
guess
for
some
really
cool
questions,
which
is
first
of
all.
A
B
B
That's
about
where
I'm
at
really
just
you
know
and
finally,
actually
making
some
videos
again.
I've
had
some
some
Flack
on
the
interwebs
I've
been
doing
too
many
conferences,
not
enough
YouTube
videos.
So
I
put
a
video
out
today
and
there'll
be
more
videos
going
now
because
I'm
actually
at
home
and
can
do
that.
But.
A
B
So
today
it
was
Unity
catalog
in
that
the
lineage
elements
finally
went
GA,
yes,
which
is
cool
because
you
can
do
column
level
lineage
now
and
you
can
click
on
a
thing
and
said:
where
did
that
come
from
and
what
table
is
that
to
go
into?
And
you
could
kind
of
force
that
to
work
in
some
other
tools,
but
it
took
like
a
huge
amount
of
work.
B
A
B
Now
you're
dancing
around
the
UK
as
well
as
over
in
Bristol
for
a
bit
in
the
taunting
I
just
yeah.
Basically,
it's
like
the
whole
lockdown
thing
of
going.
I
don't
need
to
leave
my
house
I
can
just
record
lots
of
videos
turned
into
you.
Don't
get
to
see
your
house
anymore,
this
one
I'm
not
refusing
to
leave
my
house
for
the
next
couple
of
weeks
well
except
to
go
and
do
Christmas
things,
and
you
know
drink
mulled
wine
in
London
and
hang
around
nice
things.
That's
my
plan
for
at
least
the
next
four
weeks.
B
A
Be
okay,
oh
God!
Okay,
we
actually
do
have
some
questions
coming
in,
so
let's
go
ahead
and
start
tackling
some
of
them.
So
first
question
is
from
LinkedIn
from
Paul
Wilson
and
said:
thanks
for
your
amazing
content,
so
good,
that's
great
that
you,
like
our
content,
the
world
of
spark
is
relatively
new
to
me.
A
He's
been
doing
it
for
about
six
months,
I
believe
and
it's
spark
config
along
with
repartitioning
and
clustering
is
actually
quite
complex,
and
what
are
some
of
the
top
tips
for
ensuring
that
I
always
have
the
best
configuration,
no
matter
the
data
size
and
task.
So
that's
his
context.
Any
thoughts
that
you'd
like
to
chime
in
or
anything
that
you'd
like
me
to
try
man
on
first,
your
call.
B
B
A
B
To
a
client
kind
of
that
I
hadn't
spoken
to
for
a
year
and
a
half
and
kind
of
had
a
look
and
there's
loads
of
things
that
they've
done
to
fix
some
problems
around
forcing
the
max
size
of
parquet
files.
Changing
you
know
kind
of
how
many
row
like
so
how
many
rows
in
a
row
group
all
of
the
low
level
config
you
can
do
in
the
spark
config
to
control
how
things
gets
better
and
I
was
like
a
lot
of
that
is
kind
of
fixed
from
adaptive.
B
B
B
So,
whilst
it's
intimidating,
when
you
first
start-
and
you
look
at
all
these
configuration
options
and
you
go
oh
wow,
what
what
do
I
even
put
into
these
numbers,
I'm,
usually
pretty
happy
saying
the
defaults
are
actually
okay
for
a
lot
of
it,
because
they'll
change,
because
the
engine
will
take
care
of
it
and
then
you're,
always
at
least
using
again.
As
the
question
was,
what
are
you
always
using
the
best
config
for
the
data
likes
of
no
matter
the
data
set
size
if
it
needs
to
be
able
to
handle
anything,
you've
failed.
A
Actually,
let's,
let's
let
me
add
to
that
because,
because
we're
and
we're
still
on
the
smart
conflict,
so
for
starters,
everything
of
Paul
for
everything
that
Simon
called
out
absolutely
right.
The
thing
that
I
absolutely
add
to
it
is
like
just
like
Sam
was
implying.
You
can
do
those
spark
configurations,
often
typically
at
the
cluster
level.
So
if
you're
going
to
do
them
at
least
do
them
at
the
session
level.
So
at
least
you
can
see
them
right
from
the
get-go,
but
saying
that
he's.
A
And
what
memory
I'm
using
there?
That's
that's
the
extent
of
all
I'm,
really
configuring,
because
that's
really
tied
to
your
Hardware,
like
I,
mean
what
what
what
what
how
many
cores
does
your
driver
have:
okay,
configure
accordingly,
how
many,
how
many,
how
many
nodes
executor
nodes
do
you
have
and
how
many
and
how
many
cores
and
how
much
memory
does
those
executors
have?
A
Okay,
again
same
principle
outside
of
those
things
honestly:
I'm,
not
usually
configuring
anything
unless
I
have
a
very
specific
reason,
and
since
you're,
just
starting
off
like
you,
like
you
noted,
that's
relatively
new
I,
wouldn't
bother
trying
to
go
super
deep
into
that.
Just
because
you
really
would
go
ahead
and
more
time
than
not
I'll
end
up
doing
configuration
that
basically
would
hose
yourself
in
the
future.
So,
for
example,
I
used
to
do
specific
configurations,
I
even
forget
what
those
configurations
are
now
around
data
skew
right.
A
Just
a
handle
data
skew
well
simoncall.aqe
spark
3.0
plus
includes
adapt
the
core
execution
or
aqe,
and
in
that
process
of
aqe
it
actually
solves
the
problem
of
of
data,
skew
it
actually
sub
partitions
the
data
behind
the
scenes
and
automatically
takes
care
of
that
for
you.
So
you're,
going
like
okay,
well
cool
so
because
of
that
nah
like
like
so
the
figures
I
had
was
actually
literally
fighting
with
aqe.
So
how
do
I
solve
my
problem?
Basically
to
solve
the
skew
problem?
A
I
literally
just
removed
my
configurations
and
remove
my
custom
code
and
aqe
handled
the
whole
thing
for
me
right
and
so
the
the.
If
you
really
get
into
any
details,
I
would
definitely
start
with
the
table
configurations
like
the
repartitioning
or
clustering
first,
which
we'll
talk
about
in
a
second
and
then
afterwards.
If
I
do
care
about
things,
like
maybe
size
like
the
actual
file
size
and
even
then
like
unless
you're
dealing
with
streaming.
A
Even
when
it
comes
to
batch
scenarios,
they
often
will
actually
like
basically
choose
the
right
sizes
for
you,
anyways,
so
yeah,
so
I
guess.
The
next
thing
we
want
to
do
is:
let's
go
ahead
and
talk
about
table
configurations
before
and
Paul's
actually
done
a
follow-up
question,
which
is
great.
But
let's,
let's
answer
his
question
about
tables
versus
we
go
back
to
his
next
question.
Okay,
so
yeah.
A
A
B
The
other
answer,
but
using
Delta,
you
know
so
I
used
to
do
lots
of
fiddling
around
or
trying
to
say.
Okay,
so
I've
got
my
data
frame.
I
need
to
re-partition
around
a
certain
thing,
because
I
want
to
control
how
many
files
this
is
going
to
create
I
need
to
repartition
to
a
certain
number
or
repartition
around
a
a
certain
kind
of
column
and
Spark.
You
know
try
to
really
really
work
out
how
many
files
I'm
going
to
get
so
I
can
like
control
the
size
and
all
that
stuff
and
I.
B
Just
don't
bother
anymore
partitioning
sure.
But
then,
if
I've
partitioned
and
I
end
up
with
more
files
inside
that
partition,
then
it's
really
optimal.
Well,
I'm
going
to
run
an
optimize
command
on
it
anyway.
So
I'm
going
to
clean
that
up,
I'm
going
to
worry
about
actually
taking
whatever
was
written
out
into
exactly
the
right
size
of
files
and
exactly
the
you
know
the
the
optimal
configuration,
but
that's
a
maintenance
job.
I
can
do
later.
I
don't
need
to
have
a
load
of
spark
settings.
I!
B
Don't
need
to
have
that
configuration
for
performance
inside
the
flow
of
my
data
frame
I
can
almost
separate
the
the
business
logic.
What
I'm
trying
to
do,
how
I'm
trying
to
derive
the
data
away
from
how
do
I
tweak
it
to
make
sure
it's
in
the
the
optimal
state
for
performance?
And
it's
nice
having
that
abstraction
and
going
get
the
job
finished,
get
it
down.
People
can
start
reading
it.
B
People
can
start
using
it
and
yeah
the
first
people
to
get
to
it
might
find
it's
not
the
quickest
thing
to
do,
because
it's
not
in
the
optimal
State
when
I
next
get
around
to
running
an
optimize.
I
can
clean
that
stuff
up
so
yeah
partitioning.
You
need
to
think
about
and
then
below
that
yeah
you're
talking
about
more
optimizing
you're
talking
about
you
know,
Zed
ordering
you're
talking
about
the
additional
things
you
layer
on
top,
which
don't
necessarily
have
to
do
with
what
you
do
in
an
actual
data
frame
to
get
stuff
kicked
out.
A
Yeah
exactly
so
so
again,
exactly
the
Simon
calls
Simon's
call
out.
Excuse
me
what
it
boils
down
to
is
that
there's,
when
you
use
optimize
or
optimize
Z
order
and
the
upcoming
Bloom
filter
indexes
that
will
be
included
in
Delta
OSS,
it's
already
in
databricks,
but
you
know
it's:
it's
on
the
roadmap
go
dot,
Delta,
IO,
slash
roadmap,
we're
a
little
behind
schedule
on
that
one.
A
But
the
context
basically,
is
that
it'll
restructure
and
organize
the
data
in
in
a
particular
way,
but
also
allow
for
space
filling
curves
I,
always
love
saying
that
to
basically
go
ahead
and
run
your
queries
more
efficiently.
So
that
way
you
reduce
the
number
of
false
pauses.
What
do
I
mean
by
that?
Basically,
when
spark
or
any
other
system,
that's
actually
able
to
use
Delta
is
able
to
go
ahead
and
say:
hey!
Do
I
actually
need
to
look
at
this
file?
A
That's
basically
what
it
comes
down
to
and
the
less
number
of
files
that
that
spark
actually
has
to
read
the
more
faster
it
will
go,
because
it
doesn't
actually
have
to
spin
over
additional
nodes,
additional
executors
or
waste
executor.
Just
to
even
look
hey
do
I
need
to
look
at
the
file,
basically
basic.
A
It's
basically
avoids
doing
all
that,
and
so
this
is
where
Delta
Lake
itself
will
actually
help
solve
a
lot
of
those
problems
with
the
optimize
optimizing
order,
underneath
the
covers
there's
also
things
like
when
just
or
ordinary
optimized
us,
which
is
basically
compaction,
which
is
often
when
you
run
write
files.
Maybe
you
actually
created
too
many
small
little
files.
A
All
right,
not
you
specifically,
but
me
in
the
past,
how
about
that?
Okay,
there
all
right
there
you
go
so
is
that
that
okay,
that's
what
I
would
respond.
Sorry
I
apologize
Simon,
but
the
context
nevertheless
is
with
all
of
that.
The
con.
The
idea
is
that
well,
instead,
just
run
optimize
literally
just
run
an
optimize
statement.
A
B
A
little
bit
of
nuance
on
Paul's
question
going:
will
it
will
it
partition
efficiently?
One
versus
200
versus,
like
10
000
parquet
files
is
slightly
different
because
you've
got
the
the
idea
of
partitioning
in
that
it
writes
to
separate
files,
and
then
you've
got
the
idea
of
spark
partitioning
in
which
it
creates
the
subfolders,
and
you
have
you
know
you
look
in
your
spark.
B
Folder
and
you'll
see
yeah
equals
2020
a
year
equals
2021
equals
2022.,
it's
not
going
to
partition
for
you,
you
need
to
design
a
partitioning
structure
and
as
soon
as
you've
got
data
of
a
certain
size,
you
need
to.
You
need
to
partition
that
just
for
efficiency
of
inserting
into
it
managing
it
updating
only
a
subsection,
rather
the
entire
thing,
getting
people
to
subscribe
to
change.
There's
a
lot
of
reasons.
Why
partitioning
is
nice
and
then
the
number
of
files
I
created
inside
that
partition?
B
We
don't
really
think
about
because
we
can
optimize
that
separately.
So
you
do
need
to
worry
about
partitioning.
You
don't
really
need
to
worry
about
the
number
of
files
I'm
using
data,
because
then
you
can
optimize
it
and
partition
strategy.
Favorite
participating
strategy
is
that
something
nerds
have
I've
got
one.
A
No
I
actually
I
used
to
have
a
favorite
partitioning
strategy,
but
honestly,
the
the
key
one
that
I
typically
still
do
or
just
default
to
is
literally
date
and
that's
it
and
the
reason
why
is
less
of
the
querying,
though?
Obviously
it
does
have
some
it
does.
It
definitely
has
some
context
because,
when
you're
querying
the
data,
basically
you
typically
query
the
most
recent
days.
Data
anyways
like
there's,
basically
almost
like
a
a
curve,
a
drop
down
curve.
A
So
any
of
the
older
data
that
I
can
partition
off,
I
can
go
ahead
and
literally
just
say:
forget
it
I,
don't
need
to
touch
that,
like
typically,
for
example,
when
you
talk
about
like
web
analytics,
we'll
say
that
even
if
you
accidentally
was
to
receive
data
from
anything
that's
older
than
a
month,
it
doesn't
matter
anymore.
Like
you're,
you
actually
will
no
longer
care
about
that
piece
of
information.
So
now,
of
course,
each
business
is
going
to
run
this
slightly
differently.
Okay,
so
yeah,
it's
cool.
A
B
Yeah
I
mean
same
on
our
side
right.
We
tend
to
Partition
by
date
in
the
earlier
parts
of
the
lake
automatically.
So
you
talk
about
your
bronze,
your
silver,
your
raw
base,
whatever
you
want
to
call
it
those
things,
because
that's
an
ETL
mechanism,
that's
about
going
well
take
those
updates
crash.
The
current
month
rebuild
the
month.
Whatever
update
pattern,
you
have
to
be
doing
it's
probably
date
based
the
only
thing
we
do
is
depending
on
the
size
of
it.
We
might
have
it
configurable.
B
So
it's
either
year,
year,
month
or
year
year,
month,
year
month
day,
and
that's
the
only
bit
of
advice,
I've
got
around
like
date
based
partitioning,
we
tend
to
include
the
the
hierarchy,
the
full
hierarchy
in
there.
You
know
so
yeah.
Why?
Why?
Why?
Why,
year
month,
so
you
know
2022
what
year
is
it.
A
B
My
friend
exactly
and
then
2022
12.
13
is
current
date,
so
we
can
include
the
full
date
range
and
that's
just
because
if
people
are
doing
like
range
based
queries
going
can
I
get
the
last.
You
know
30
days
because
I
get
older
between
those
two
points.
It's
really
awkward
to
build
that
SQL
query
when
it's
can
I
take
that
year,
but
this
day,
if
it's
past
that
point
in
a
month
from
that
point
in
a
month-
and
you
end
up
building
this
complex
thing
to
try
and
hit
those
budgeting
things.
A
A
You
know
as
much
as
I
would
love
to
debate
him
for
the
sake
of
being
a
North
American
he's
actually
completely
right
and
I've
been
doing
the
exact
same
thing
for
her
as
he's
as
he's
suggesting
for
years
like
this
is
not
even
like
specific
to
spark
I've
been
doing
this
We've.
We've
we've
been
doing
this
in
SQL
server
and
everything
else,
just
for
the
exact
same
reason,
so
yeah.
A
We
didn't
Rat
Hole,
like
we
usually
do
yeah,
okay,
so
there's
a
question
on
the
line
here
from
the
Q
a
panel
here
in
the
zoom
side
of
the
house,
so
we're
going
to
switch
over
to
that
by
the
way
for
those
folks
in
LinkedIn
and
YouTube
If.
You
still
have
any
other
questions
go
ahead
and
please
chime
in
otherwise
that
will
wrap
it
up
because
we
don't
want
to
make
you
guys,
suffer
and
listen
to
my
awful
voice.
A
So
but
the
question
is
and
in
data
bricks
the
first
time
you
run
an
automated
ETL
job
and
the
target
table
does
not
exist
yet.
Do
you
script
the
table
Creation
in
the
same
script,
use
terraform
or
do
something
else,
I'm
looking
for
Best,
Practices,
okay,
so
that's
a
great
question
and
there's
actually,
unfortunately,
I
believe
Simon
and
I
can
rat
hole
on
this
one
as
well.
So
do
you
want
to
start
first
or
do
you
want
me
to
start?
First
on
the
run
all
in
here.
B
Mean
I'll
say:
there's
two
there's
two
answers:
there's
the
there's
the
brand
spanking
new
answer,
which
is
what
I
think
people
are
gonna
start
looking
at
and
there's
the
answer
of
how
we
currently
do
it
now,
the
majority
of
times
when
we're
talking
about
tables,
we're
doing
a
merge
and
you
can't
merge
into
a
table
that
doesn't
exist.
Therefore,
yes,
you
have
to
create
the
table
in
advance.
It's
annoying
I'd
love.
The
merge
function.
I
want
to
go.
Oh
it's
not
there,
I'll
just
create
it
for
you.
B
That
would
make
my
life
so
much
easier,
but
it
doesn't
so
yeah.
Absolutely
we
ended
up
writing
a
little
table.
Creation
function.
That
goes
basically
builds
up.
A
table
creation
script
in
spark,
SQL
and
goes
you
know,
create
table
if
it
doesn't
exist.
Here's
all
the
things
now
that
felt
like
a
bit
of
a
work
around
the
hacky
thing
originally,
because
what
we
wanted
is
our
tables
just
to
be
created
based
on
the
contents
of
the
data
frame.
B
So
I
can
just
hydrate
an
entire
Lake
and
go
generate
a
thousand
tables
you'll
figure
out
what
the
schema
is
when
you
receive
the
data
and
then
just
create
the
tables
according
to
what
you
got
and
that's
that's
how
we
used
to
work
around
things
but
as
Delta's
evolved
to
the
point
where
you've
got
loads
of
things.
That
can
only
really
be
done
on
the
table
creation
step
when
we're
creating
you
know:
Bloomfield
indexes
we're
creating
identity,
columns,
we're
putting
constraints
in
we're,
putting
comments
on
the
various
columns.
B
All
those
various
different
things
need
to
be
done
as
part
of
the
table
creation
script,
and
so
there's
the
need
for
that
initial
table
creation
function.
Actually
you
have
to
create
the
table
up
front
so
that
actually
can
be
done
properly
and
start
to
get
some
of
these
more
advanced
modeling
aspects
that
are
being
enabled
by
Delta.
So
whilst
it
was
a
necessity
because
we
wanted
to
merge
originally
now
it's
something
where
we
can
put
in
and
start
to
harness
a
lot
loaded,
the
extra
Advanced
features
by
taking
advantage
of
it.
B
So
that's
kind
of
where
we
are
now
going.
Yeah
yeah,
we
create
a
function.
We
just
whenever
you're
about
to
merge
into
it
when
everybody
to
write
into
it,
call
that
function
first
build
up
a
script
that
says,
create
the
table.
If
it
doesn't
exist,
add
in
any
of
those
additional
column
level,
functionality,
things
you
can,
and
you
know
Life's
good,
then
there's
the
what's:
what's
new?
What's
what's
recently
changed,
and
this
makes
all
of
the
old
school
SQL
people
so
happy
in
that
I've
not
had
a
go
of
it.
B
Yet
I've
not
tried
it,
but
Irwin
data
modeler
is
now
on
Partner
connect,
so
you
can
actually
build
out
your
proper
crow's
feet.
Data
model.
You
can
have
it
all.
You
know
you
can
fill
your
data
modeling
desire,
make
it
look
as
old
school
as
you
like,
then
you
can
hit,
go
and
generate
the
SQL
and
create
those
tables.
So
there's
kind
of
like
the
people
who
used
to
build
lots
of
warehouses
and
have
a
data
model
deploy
that
data
model
and
then
configure
their
pipelines
to
fill
that
data
model
can
work
that
way.
A
Before
I
think
I
think
so
so
by
the
way
to
give
provide
people
context,
so
it
for
many
of
the
newer
people
into
the
data
warehousing
space.
Now,
when
I
say
data
housing
I'm,
not
actually
talking
about
the
tech
itself,
I'm
actually
talking
about
the
techniques.
Okay.
So
when
we
like
a
lot
of
people,
talk
about
data
warehouse,
the
the
software
itself,
like
SQL,
Server
data
warehousing,
that's
just
just
because
both
of
both
Simon
and
I
are
Microsoft.
These
all
used
to
be
microsofties
or
whatever
close
enough.
A
So
the
so
we
we
can
talk
about
the
data
house,
warehousing
the
actual
technology
or
software
service,
or
we
can
talk.
Dave
has
the
techniques,
the
methodologies,
and
so
what
we're
talking
about
in
this
case
is.
Is
the
methodology
not
the
actual
software
itself,
so
because
we're
talking
in
that
context,
Erwin
in
the
old
days?
Okay,
as
in
when
I
started
working
in
data
warehousing,
both
the
technique
and
the
software
back
in
like
the
late
90s,
so
I'm
very
much
dating
myself
now
by.
A
No
true,
but
the
point
is
that,
like,
if
you're,
if
you're
madly
into
doing
data
warehousing
techniques,
then
Irwin
is
your
absolutely
what
you
would
go,
do
you
would
design
the
system
come
completely
go
Bonkers
and
exactly
what
Simon
was
calling
out.
You
have
this
wonderful,
logical
and
then
physical
layout,
of
what
those
tables
are
supposed
to
look
like
what
the
primary
key
and
foreign
key
relationships
to
them
are.
What
indexes
columns
all
these
other
things
are
boom,
and
then
you
go
and
say
all
right.
A
Now,
click
boom
all
these
SQL
carriers
are
generated
for
any
of
these
databases
and
Bam.
Now
you
have
your
wonderful
data
warehouse
with
created
by
multiple
tables,
and
this
wonderful
star
or
snowflake
schema
that
you
know
that
that
was
Perfection
and
then
absolutely
as
soon
as
you
did
it,
then
you
would
change
half
of
it
because
of
whatever
reasons
right.
A
But
the
context
basically
is
that,
if
you're
able
to
plan
things
out,
the
this
is
a
very
powerful
tool
that
you
can
basically
leverage
part
of
the
reason.
We've
not
leveraged
something
like
this
isn't
because
of
its
in
availability,
though
part
of
it's
because
of
the
fact
is,
although
that's
true
part
of
it's
just
because
of
the
fact
that
we
ourselves
are
typically
because
we're
running
so
fast
exactly
to
what
Simon
called
out.
A
I
meant
to
merge
another
table,
and
so
it
was
basically
that
debate
back
and
forth
whether
it
made
sense
or
not.
To
actually
address
that,
and
so
yeah
I
mean
the
like
I
said:
I'd
probably
go
create
an
issue
specifically
to
address
that,
but
not
either
way,
as
Simon
called
out
the
reality
for
the
the
type
of
scenario
when
it
comes
to
like
what
do
you
want
to
do?
A
For
example,
if
you're
a
terraform
script
expert
yeah,
why
not
just
spin
up
a
terraform
script
that
actually
defines
the
tables
from
the
get-go?
If
anybody
who's
on
this
session
today,
you've
never
heard
of
terraform.
No,
don't
do
that.
Dude
build
a
build,
a
function
instead
that
you're
more
familiar
with,
and
that
honestly,
is
your
best
practice.
The
key
thing
and
I
think
everybody.
If
you're
gonna
get
anything
out
of
this
is
to
document
it
write
it
down
somewhere,
so
people
will
recognize
the
fact
that
you're
doing
this,
please.
A
A
Yeah,
exactly
yeah
I
was
about
to
say
I'm,
like
I,
hope,
you're
being
sarcastic
here,
because
yeah,
okay,
okay,
good
good,
all
right,
all
right,
cool,
there's
another
question
about
the
databricks
ecosystem.
So
normally
we
don't
typically
answer
too
many
of
them,
but
is
it
isn't
ask
us
anything
so
we'll
try
to
actually
make
it
shift
back
to
also
talking
about
the
wider
Delta
ecosystem
as
well?
But
it
is
a
great
question
from
Ricardo
on
LinkedIn.
So
let's
go
ahead
and
Tackle
this
one,
okay,
so
Ricardo
his
question
is
I've.
A
A
Basically,
we
have
spark
connect
coming
out
really
soon
and
I
would
100
start
giving
that
a
try.
We
actually
will
have
a
a
separate
session,
specifically
for
spark
connect
early
next
year
to
talk
about
that.
This
will
allow
for
better
IDE
integration.
Basically,
now
saying
that
there
already
is
a
vs
code,
plugin
okay,
that
was
released,
I
want
to
say
announce,
though,
actually
last
week,
I
actually
don't
quite
remember.
A
Okay,
thank
you,
I'm
like
why,
because
I
know
Gerard
broccol
from
Pico
right,
he
had
him
and
his
team
had
created
an
initial
vs
code,
plug-in
I
want
to
say
a
month
and
a
half
two
months
ago,
and
then
it
wasn't
on
purpose.
Databricks
actually
had
been
working
on
one
in
parallel,
but
the
context
basically
nevertheless,
is
that
yes,
there,
whether
it
is
talking
about
the
open
source,
spark
ecosystem,
I
love,
spark
connect
or
we're
talking
about
the
databrick
specific
ecosystem.
A
Both
sides
to
basically
have
much
much
better
integration
with
existing
Ides
and,
if
you're
wondering
why
vs
code,
it
is
actually
literally
the
number
one
IDE
right
now
like
unequivocally
I
I
cannot
explain
to
you
like
how
weird
it
is
for
me
to
say
that,
because
I
used
to
use
Visual
Studio
itself,
where
I'm
and
I'm
old
school,
so
that
means
I
had
like
you
know
these
20
DVDs
that
I
had
to
put
into
my
machine
to
go.
Install
just
visual
c-sharp.
A
Done
exactly
yeah
yeah,
pretty
pretty
much
yeah
yeah
I
I
yeah
I
hated
it
that
much
apparently
so,
but
never.
Okay!
Sorry,
as
you
can
tell
a
traumatic,
traumatic
experiences
from
my
past,
but
nevertheless
the
context
is
that
there
is
a
lot
more
work
being
done
for
integration
with
CI
CD
platforms,
get
integration
definitely
already,
and
basically
just
integration
within
the
ID.
Anything
else
that
you
think
I
I
forgot
to
add:
Simon
nope,
nothing
like.
B
All
makes
sense
to
me.
The
other
thing
is
I
was
in
in
Vegas
when
I
saw
the
announcement
for
the
the
vs
code
extension.
Is
it
just
for
the
database
SQL
side
of
things?
So
it's
like
interacting
with
the
SQL
endpoints
and
the
SQL
data
warehouses,
or
is
it
just
cover
the
board
of
lots
of
different
things?
I've
not
actually
looked
into
it?
I
haven't
either
all
right
question
answered,
we
don't
know,
I
am
assuming
I'll
have
to
do
a
video
on
that
very
soon.
I'll.
A
Go
yeah
yeah,
so
so
Simon's
gonna
do
a
video
on
that
soon.
To
answer
his
own
question,
because
since
I'm
usually
sitting
in
the
Delta
lake
side
of
the
house,
as
it
is
anyways
I
I
know
it
sounds
funny,
I
realize
I
am
a
datarix
employee,
but
it's
really.
This
is
hilariously.
Funny
part
of
the
reason
I
have
Simon
here
is
because
he
can
actually
answer
the
databrous
questions.
So
I
often
don't
know,
which
is
a
horrible
thing
to
say,
as
a
database
of
play.
I
do
want
to
call
that
out
so
ends
all
right.
A
We
do
have
another
question,
so
we
can
shift
to
so
that
way.
I
can
avoid
what
I
just
said
in
a
DLT
pipeline.
This
is
data.
Delta
live
tables
in
the
UI.
You
define
a
Target
DB
schema,
so
a
table
defined
as
raw
would
go
to
Target
raw
okay,
fair
enough
can
I
define
the
next
DL
table
DLT
table
in
the
pipeline.
That
would
land
in
a
silver
clean,
for
example,
yeah
I
mean
so
Jonathan.
A
B
Yeah
I
think
we
had
a
similar
question
last
time
in
that
it's
they
want
to
populate
different
databases,
the
tables
into
different
databases
using
DLT,
and
you
can't
currently
do
that
right.
It
would
go
into
a
single
catalog
in
your
meta
store
and
you
just
get
different.
You'd
you'd
have
to
currently
label
it
as
raw
table
silver
table.
You
need
to
prefix
those
names
at
some
point
right.
A
But
but
it
wouldn't
go
into
a
separate
database.
It
could
only
go
in
the
same
one
in
essence,
quote-unquote,
but
the
idea
is
that
you
could
label
them
accordingly
Jonathan.
This
is
definitely
one
of
the
things
where
we
would
advise
you
to
definitely,
and
we
will
help
too
so
don't
get
me
wrong,
but
to
go
ahead
and
definitely
ping
your
databricks
essay
and
then
basically
say
yeah.
We
should
be
supporting
that
now,
because
we're
actually
none
of
us
are
we're
all
in
agreement.
Basically
I
mean
yeah
like
yeah.
That's
it
like.
B
B
Unity,
catalog
and
Delta
live
tables
can't
talk
to
each
other.
So
eventually,
you'll
need
three
part
naming
in
those
targets,
but
they
they
don't
currently
exist,
I'm
sure
when
they
do.
Hopefully,
team
they'll
be
looking
at
that
to
figure
out
how
those
things
talk
to
each
other
but
yeah.
Currently,
it's
a
figure
out
naming
convention
and
it
all
lumps
into
a
same
database
for
now.
A
Yeah
for
now,
for
now,
but
again
Jonathan,
please
do
ping
we're
we're
glad
to
to
back
you
up
on
that
one
easily
enough
cool.
All
right!
Well,
I!
Think!
That's
it
for
questions,
anything
else
that
you
want
to
add.
Otherwise,
we'll
cut
it
short
for
today,
but
though
it's
still,
it
still
was
37
minutes.
So
it's
not
too
bad
for
us.
So.
A
Okay,
like
we're
just
waiting
scene-
oh
okay,
we
do
actually
have
more
questions.
Okay,
cool,
all
right,
all
right!
So
then
we're
gonna!
Ask
anybody.
If
you
have
questions,
please
put
them
down,
because
both
Simon
and
are
lazy,
so
we
actually
don't
mind
leaving.
So
if
you
have
questions
go
ahead
and
chop
them
in
here
right
now,
so
that
way
we
will
actually
address
them.
Okay,
I
need.
A
Exactly
that's
what
I'm
saying
I
mean
yeah
like
you
know:
I
gotta
go
make
coffee
so
like
we
have
priorities
here
like
really
it's
yeah.
So
oh
okay,
all
right,
but
let's
I
was
about
to
go
into
a
dietrop
of
coffee,
but
I
just
realized.
That's
not
that's!
Not
something!
I
should
do
so.
Let's
go
ahead
and
ask
pavel's
next
question.
A
Thanks
for
Delta
is
great
yeah
glad
to
hear
that
oh,
except
see.
Actually,
everything
on
the
lake
house
side
is
a
pleasure
to
work
with.
How
are
the
challenges
to
get
the
data
to
Lake
AK
ingestion
into
the
tier
one,
I.E
bronze,
I.E
raw
okay?
Is
there
any
best
practices
for
multi
for
a
multi-tenancy
model?
How
do
you
do
the
same
ingestion
by
2,
000.,
Okay,
Pavel
I
want
to
answer
your
question
I'm
going
to
start,
but
I
also
want
I'm
going
to
start
by
actually
asking
you
some
questions.
A
A
That's
really
what
it
comes
down
to
the
whole
reason
spark
became
so
popular.
Was
this
idea
that
I
could
have
linear
distribution?
Add
more
nodes,
can
process
more
data
or
query
more
data?
That's
what
it
boils
down
to
so
unequivocally.
I
would
start
with
that,
which
is
saying,
is
whatever
structure
you
have
I,
don't
care
which
managed
platform
or
which
OSS
platform.
If
you're,
spinning
the
doing
a
DIY
or
spinning
up
yourself,
are
you
able
to
go
ahead
and
actually
be
able
to
say?
A
Let
me
go
ahead
and
now
go
from
10
nodes
to
50
to
200.
Okay,
that's
the
first
thing.
The
second
thing
is,
then,
when
you
do,
that
is
the
underlying
infrastructure
that
you're
trying
to
do
actually
able
to
support
that
now.
For
example,
if
I'm
doing
this
in
the
cloud,
presumably
like
I'm
just
going
to
choose
AWS
just
because
I
was
working
on
that,
like
literally
before
we
started
this
through
a
session,
but
the
same
thing
is
just
for
HD
and
HD
insight
for.
B
A
Know
I
know
sorry,
I'm,
sorry
trauma
trauma's
coming
back
again:
okay,
so
for
Azure
or
for
GCS.
Okay,
the
content,
the
the
context
is
basically
that
you
could
basically
have
S3,
because
it's
a
specific
for
AWS.
So
literally
you
add
prefixes
to
the
the
file
system
in
order
to
be
able
to
basically
help
with
the
distribution,
and
so
now
S3
is
actually
able
to
send
back
the
information
just
as
quickly
as
your
system
is
about
to
live,
beat
The,
Living
Daylights
out
of
it,
which
is
great.
A
So
if
you
have
like,
for
example,
you're
doing
this
as
an
on-prem
environment.
My
little
response
is
that's
great.
Well,
hdfs
can
actually
handle
it,
but
make
sure
you
actually
have
a
direct
router
link
between
whatever
your
compute
and
whatever
is
storing
storage,
because
otherwise
you're
going
to
do
the
equivalent
of
the
DDOS
attack
on
your
own
own
systems.
Okay,
I
I
am
guilty
of
that
very
very
guilty
of
that.
So
so
don't
do
that.
But
so
in
the
end
it
should
presumably
be
about
that.
A
Okay,
which
is
to
say
like
okay,
can
I,
have
a
can
I
spin
up
spark
and
can
I
increase
the
number
of
nodes
to
it
and
is
the
underlying
cloud.
It
was
underlying
infrastructure
going
to
be
able
to
handle
it
so
because
of
that
I'm
presuming
Cloud
now,
okay,
which
then
okay,
that's
great,
which
means
it
should
be
able
to
handle
it.
Provided
you
you
can
get
enough
spot
or
regular
instances
to
be
able
to
basically
query
against
that
now.
A
Another
aspect
of
multi-tenancy
is
basically
the
idea
that
okay
am
I,
going
to
be
able
to
and
I'm
going
to
pick
on,
AWS
now,
mainly
because
there's
actually
a
weak
point,
there
now
I'm
going
to
do
that,
because
if
I
want
multiple
drivers,
multiple
clusters
to
basically
go
ahead
and
write
to
the
same
Delta
table
at
the
same
time.
Okay,
so
that
works
perfectly
fine
in
azure.
It
also
works
perfectly
fine
in
GCS
and
also
works
perfectly
fine
in
hdfs.
A
A
Okay,
let's
say,
say
metadata
at
the
exact
same
time,
there's
a
possibility
that
client
2
will
overwrite
client
one,
thus
corrupting
the
system.
Now,
of
course,
you're
going
like
oh
my
God.
This
is
that
it
is
except
okay
in
databricks
how
you
solve
that
problem
is
that
datrix
already
has
an
S3
commit
service,
that's
already
included
and
we're
currently
working
on
making
that
open
source
by
the
way.
A
So
that
way,
any
client
that
interacts
with
a
databricks
environment
can
actually
go
ahead
and
access
it,
not
just
databricks,
SQL
or
databrick
spark
or
whatever
any
client.
That's
interacting
with
databricks
actually
can
make
use
of
that.
But
then
you
ask
hey:
how
do
I
do
the
open
source?
Well,
we've
been
we've
been
doing
this
for
a
while.
Actually
since
Delta
Lake
1.2,
the
is
the
S3
dynamodb
log
store,
basically
dynamodb
acts
as
a
conditional
right.
A
So
that
way
when
client,
one
and
client
two
are
both
trying
to
write
to
S3
at
the
exact
same
time,
the
log
itself
at
the
same
time,
basically,
what
ends
up
happening
is
like
oh,
okay,
dynamodb
says:
yes,
you're
allowed,
writing
it
and
no
you're
not,
and
the
writing
actually
isn't
even
writing
the
actual
Json
file.
The
Json
files
are
actually
appended
with
a
with
the
prefix
guid
and
a
temp
file
like
attempt
naming
invention,
so
they
actually
don't
ever
override
each
other.
A
A
So
how
often
does
this
occur
not
often
by
the
way,
even
though
I'm
telling
you
about
all
of
the
stuff
when
we
did
the
tests
I
want
to
say
about
three
years
ago
now,
four
years
ago
now
there
was
four
chances
out
of
10
million
that
it
could
happen
Okay.
So
it
is
not
exactly
a
common
occurrence,
but
because
we
wanted
to
guarantee
this
idea
of
multi-tenancy
multi-transaction
multi-drivers.
A
That's
why
we
built
again
in
databricks
the
S3
commit
service
and
in
open
source,
the
Delta
World,
basically,
the
S3
dynamodb
log
store,
and
so
that's
actually
being
built
for
other
systems
as
well.
So
I'm,
Pablo,
I,
hope,
I've
answered
your
question
because
again,
multi-tenancy
can
mean
different
slightly
different
things,
but
at
least
I
believe
I
answered
mostly
the
the
cloud
related
and
the
multi-driver
related.
So
anything
else
that
you
think
I
can.
That
needs
to
be
added
here
from
a
die.
Try
perspective
Simon,
I.
B
B
If
you
need
to
do
separation
of
storage
for
your
multi-tenancy,
but
we
just
treat
that
as
like
that
integration
layer
right,
so
it
doesn't
have
any
rules
they
can
throw
whatever
files
in
there.
It's
not
going
to
go,
that's
the
wrong
structure
and
throw
things
back.
It's
deliberately
a
decoupled,
easy
Landing
space
for
people
with
profiles,
and
then
we'll
tend
to
file
streaming
into
that
first
raw
area,
especially
if
you're
using
daily
Works
we're
using
things
like
autoloader,
so
that
we're
then
working
on
going
from
that
Landing
space
into
there.
So
you're
not
worrying
about.
B
B
Various
different
ETL
tools
are
good
at
getting
data
and
land
a
files
somewhere
and
then
from
there
is
where
I
tend
to
kick
off
the
spark
jobs.
So,
honestly,
I
don't
have
to
worry
about
all
the
things
that
you
were
mentioning
Danny.
That's
like
you
know,
just
hey
give
me
some
files
and
I'll
work
with
the
files.
If
I
have
to
go
and
get
the
files
myself,
I'll
just
use
a
tool
that
is
for
moving
files
rather
than
for
doing
the
big
chunky.
Cloud-Based
queries
that
are
spark
is.
A
A
Cool
all
right
so
actually
say
this.
This
is
a
good
way
of
time
for
us
to
wrap
up.
I
did
want
to
do
two
two
little
bit
two
call
outs
eventually,
I'll
be
able
to
speak
English.
First
thanks
to
Chris
frankley.
He
actually
went
ahead
and
called
out
the
the
blog
post
that
I
was
actually
talking
about
in
terms
of
the
S3
dynamodb
lockstar,
so
I
propped
it
into
into
the
chats.
A
A
This
is
a
LinkedIn
video,
so
it
actually
stays
in
the
Delta
Lake
LinkedIn,
so
you
can
always
come
back
to
it.
The
other
thing
to
call
out
is
that
we
also
prop
them
in
the
Delta
Lake
YouTube,
which
is
go.delta
dot,
IO,
slash
YouTube,
so
you
can
just
go
there,
there's
actually
a
playlist
called
Simon
and
Denny's
AUA,
so
yeah
by
all
means
go
ahead
and
you
can
watch
these
videos
and
make
fun
of
us
if
you'd
like
so
saying
that
I
think
that's
it.
A
B
A
For
example,
Chris
fragley
has
not
hinted
that
the
we
should
start
having
guest
appearances.
You
know
what
I
think.
That's
a
great
idea
because
and
in
fact
Chris
ping
me.
We
should
figure
this
out.
Chris
frankley
is
an
old
friend,
so
so
I
apologize
that
we're
just
reconnecting
online
like
this.
But
yes,
absolutely
yeah
yeah.
You
thought
you'd,
never
ask
okay,
so
please
go
do
chime
in.