►
From YouTube: Tech Chat: BI to Lakehouse Round 2 - Electric Boogaloo
Description
Considering shifting gears into Spark Data Engineering, BI, and Lakehouses? Join this fun session with Simon Whiteley (@mrsiwhiteley) and Denny Lee (@dennylee) as we CONTINUE to chat through their meandering journeys from SQL Server & BI to Apache Spark, Delta Lake and the emerging Data Lakehouse approach. Be prepared for a geeky, trans-Atlantic event from two data nerds....this time we'll actually answer your questions.
Link to watch round 1: https://youtu.be/ls7-LKwKFYA Databricks is proud to announce that Gartner has named us a Leader in both the 2021 Magic Quadrant for Cloud Database Management Systems and the 2021 Magic Quadrant for Data Science and Machine Learning Platforms. Download the reports here. https://databricks.com/databricks-named-leader-by-gartner
A
For
the
session
today,
awesome,
thank
you.
Yeah
hi,
I'm
simon,
hello,
yeah.
So
there's
a
few
good
questions
in
the
comment
going.
What
is
this?
So?
I
guess
less.
Maybe
we
didn't
make
that
like
that,
that
clear
so
yeah
my
name
is
tom
wiley,
I'm
from
the
uk,
I
run
a
consultancy
and
I
do
lots
of
things
around
databricks
spark
data
lakes,
data
engineering.
All
of
that
kind
of
stuff.
A
B
Perfect,
I
think
you
covered
it
extremely
well,
so
my
name
is
denny
lee,
I'm
based
in
seattle.
Washington,
though
I
originally
am,
for
the
person
on
linkedin
live
that
noted
that
he's
from
halifax
I'm
actually
originally
from
newfoundland.
Yes,
I'm
not,
I
don't
realize
I'm
not
the
most
obvious
newfie
around,
but
yes,
I'm
actually
a
maritimer
as
well.
Originally
a
little
bit
my
history
exactly
what
simon
called
out.
We
want
to
answer
lots
of
your
questions
when
it
comes
to
become
all
sparky
data
engineery
right.
B
B
What
happened
was
that
I
end
up
going
to
mcgill
university
of
montreal
specifically
to
become
a
doctor
and
then
that's
where
I've
learned
computer
engineering
and
then
here
we
go
so
I
end
up
joining
microsoft
as
a
I
think.
What
web
developer
even
like
yeah?
That's
now,
I'm
aging
myself
really
nicely
here,
but
in
the
process
of
doing
all
this
end
up
joining
the
sql
server
team.
So
I'm
actually
an
old
database
guy
was
on
the
team
that
helped
build.
B
What
you
now
know
is
hdinsight,
so
both
either
you
can
say
thank
you
or
I
can
apologize,
whichever
one
you
want
to
take.
That's
completely
cool
with
me
and
then,
of
course,
now
I'm
currently
at
databricks
as
a
developer,
advocate
loved
working
with
spark
early
on,
so
they
were
silly
enough
to.
Let
me
join
the
company,
so
here
we
are
same
idea,
lots
of
cool
questions
and
actually
we've
got
a
great
question
right
from
the
get-go.
I
figured
this
would
happen
right
from
raymond.
What
happened
to
my
arm?
Yes?
B
Well,
I'm
an
idiot,
that's
what
happened
actually,
and
so
the
quick
call
out,
for
example,
is
that
not
example.
Excuse
me
was
that
I
decided
to
re-aggravate
my
rotator
cuff
intrigue
now.
Why
would
you
say
why?
Why
did
I
do
that?
Well,
it's
because
I
decided
to
go
and
do
some
pt
over
the
weekend,
and
but
except
I
did
the
pt
watching
american
football.
The
seahawks
was
having
a
tough
game
with
the
minnesota
vikings,
our
run.
Defense
wasn't
working
very
well,
so
it
was
very
stressful.
B
So
what
was
supposed
to
be
only
10
to
15
minutes
of
pt
resulted
in
about
three
hours
of
pt.
I
celebrated
after
we
won
and
then
realized
wow.
My
arm
hurts.
So
here
we
go.
So
that's
what
happened
and
now
I'm
currently
wearing
a
sling
to
compensate
for
my
idiocy.
So
there
we
go,
oh
and
by
the
way
before
we
get
to
the
rest
of
the
questions
I
did
want
to
call
out.
We
want
to
thank
karen
for
returning
from
her
from
maternity
leave.
B
B
A
Being
able
to
talk
via
active
directory
is
a
huge
thing
for
security
being
able
to
do
the
v-net
injection
in
terms
of
digital
architecture.
That's
that's
a
huge
one
for
getting
it
to
be
secure
and
all
things
like
private
link
and
all
that
kind
of
stuff.
Anything
that
we're
talking
about
in
the
azures
kind
of
thing,
it's
all
to
do
with
how
it
plugs
into
the
esoteric
things
to
do
with
azure
so,
and
that
tends
to
become
the
main
core
differences
that
I've
seen.
A
We
don't
we
don't
yet
in
a
jawline,
and
they
announced
that
it's
coming
later
this
year,
which
is
cool,
but
that's
the
kind
of
thing
that
we're
looking
at.
So
it's
less.
What
does
the
actual
engine
the
bugs
do
it's?
How
does
this
whole
thing
talk
to
the
cloud
platform
tends
to
be
the
differences
that
I
see.
B
Perfect
and
so
just
to
add,
chime
myself,
and
since
I
I
came
from
both
the
data
bricks,
sorry,
the
azure
perspective
and
also
have
the
aws
perspective
dear.
It
really
is
about
which
ecosystem
you
or
ecosystem.
You
want
to
use
right,
and
I
I'm
there's
a
little
play
on
words
in
terms
of
echo
system
versus
ecosystem
is
they're,
it's
the
same
word
but
just
said
it
in
a
different
way.
B
Obviously
right,
but
the
context
is
in
a
lot
of
ways:
the
clouds
are
in
a
lot
of
the
same
way
right,
obviously,
the
some
of
the
programming
styles
for
the
azure
echo
system,
you,
if
you're
more
used
to
net,
for
example.
Obviously
that
azure
makes
a
ton
more
sense
right,
but
each
both
systems
have
their
own
version
of
integrations
right.
They
both
have
a
cloud
storage
system,
some
type
like
you
know,
s3
versus
adls,
gen,
2,
right
they're,
both
meshgrid
networks
right.
So
I've
got
lots
of
stories.
B
In
fact,
maybe
I'll
go
into
it
later,
but
a
lot
of
stories
about
what
happens
when
you
don't
have
a
meshgrid
network.
Okay.
So
in
terms
of
collapsing,
your
saturating,
your
network,
basically
but
the
the
point-
is
that
you
know
each
one
of
them
has
a
bunch
of
really
interesting
mechanisms
of
different
pass
services,
vm
services.
So
it
really
depends
on
which
one
you
really
are
plugged
into
more
and
in
fact,
for
many
people
or
many
customers
for
that
matter,
they're
actually
plugged
into
both
right.
B
They
have
both
aws
and
azure,
and
for
that
matter
the
other
clouds
as
well
just
because
they
want
to
have
a
multi-cloud
solution
right
and
and
there's
pros
and
cons
to
each
approach
right.
So,
for
example,
if
you
decide
that
you
only
want
to
use
the
aws
ecosystem,
there's
the
advantage
of
saying
now,
I
can
just
integrate
directly
with
working
with.
B
B
Maybe
love
is
a
little
too
strong,
but
lots
of
people
like
really
like
using
aed
right
in
terms
of
identity,
security,
yeah
and
all
completely
valid,
completely
true,
and
so
often
more
times
than
one
you'll
end
up,
leaning
to
one
ecosystem
because
of
a
set
of
features.
That
makes
more
sense
for
you
right
and
then
also
where
the
multi-cloud
kicks
in.
Is
that
it's
because
first
and
we
have
at
least
there's
plenty
customers
that
do
this
like
where
they
actually
have
aad.
B
So
that's
why
that's
the
reason
why
they
start
off
with
the
azure
ecosystem,
but
they
loved
a
particular
aws
set
of
components
too.
So
at
least
when
it
comes
to
databricks
we're
like
yeah,
we
work
on
both,
so
it
doesn't
matter
right.
So
you
do
you.
Do
you
right?
You
do
what
you
think
is
best
for
you
and
so
we're
not
really
leaning
you
one
direction
to
the
other
there.
B
A
A
Has
the
same,
certainly
in
azure
there's
a
cost
for
taking
data
out
of
a
job,
it's
free
to
put
data
into
as
young,
but
if
you're
taking
it
out
of
one
of
the
azure
date,
centers
of
moving
it
somewhere
else,
there's
an
egress
cost.
So,
if
you're
like,
if
you
build
data
breaks
in
like
so
for
enable
west
over
there
and
you've
got
all
of
your
source
data
and
your
terabytes
and
terabytes
still
save
over
there
and
you're
pulling
it
out,
it's
just
gonna
cost
you
my
money!
A
B
Exactly
no,
no
some
you
hit
it
dead
on,
I
mean
most
of
the
customers
that
actually
have
to
deal
with
multi-cloud
environments.
That's
exactly
what
they
do.
They
basically
process.
The
data
have
a
minimal
amount,
that's
sitting
that
needs
to
be
transferred
from
one
environment
to
the
other
right
if
they
for
state
government
need
to
transfer
the
data
from
aws
to
azure,
or
vice
versa,
or
for
that
matter,
like
I
said,
the
other
clouds
right.
You
know
they
need
to
transfer
to
google
cloud
structure
or
whatever
else
yeah.
B
They
try
to
minimize
the
amount
of
data,
and
so
often
what
we'll
see
our
customers
do
is
they'll
they'll.
Take
it
make
use
of
databricks
to
basically
aggregate
the
data
and
be
prior
to
transfer
of
the
data
right.
So
this
is
a
very
good
approach
to
minimize
the
overall
cost.
Basically
so
yeah
anyways
yeah,
I
think
we
covered
it
anything
else.
You
think
we
should
add.
A
C
B
Okay,
cool,
let's
dive
to
the
next
question,
because
it
looks
like
at
least
from
linkedin
live
right
now
looks
like
there's
a
lot
of
questions
and
we
got
some
questions
from
zoom
and
youtube.
So,
let's,
let's
go
through
the
next
set
of
questions,
so
we
can
start
answering
those
questions
too,
all
right.
The
next
question:
what
is
the
difference
between
lake
house
and
delta
lake
right?
Because
we
did
talk
about
lake
houses?
So
did
you
want
to
start?
First?
You
would
like
me
to
start
first
simon.
You.
B
You
go
for
the
official
database
answer:
okay,
cool
I'll
go
for
the
official
database,
so
the
official
databricks
answer
would
be.
Lake
house
is
an
architectural
pattern
or
you'll
often
hear
you
know
the
marketing
for
phrasing,
an
architectural
paradigm,
okay.
So
it's
a
pattern
that
brings
a
power
reliability
of
traditional
data
warehouse
functionality
combined
with
ai
and
ml
to
cloud
data
lakes.
B
B
There
are
just
because,
when
we
made
that
shift
to
data
lakes,
which
was
a
good
shift
by
the
way,
I'm
not
saying
it
was
a
bad
chef,
but
we
forgot
to
do
a
lot
of
things,
and
this
is
coming
from
a
sequel,
hound
and,
and
I'm
sure
simon
will
toss
in
lots
of
comments
on
his
own
on
this
one
right.
We
forgot
a
lot
of
things.
We
forgot,
for
example,
simple
things
like
what
is
your
model
of
your
data.
B
We
kept
on
getting
obsessed
with
this
idea
of
scheme
on
rings,
scheming
on
rescue
and
yeah,
we'll
figure
it
out
later,
except
now.
What
you
have
is
a
lot
of
noise
as
opposed
to
data,
a
lot
of
there's
little
actual
insider
information
coming
from
it
because
we're
so
busy
shoving
in
there-
and
it
doesn't
mean
data
lakes
themselves-
are
a
bad
idea.
It's
just
more
that
we
got
inherently
lazy.
B
We
forgot
to
apply
software
engineering
practices
to
this
when
we
said
scheme
on,
read
and
so
the
whole
context
to
say,
wait
what
were
the
advantages
of
database
technologies
or
data
warehouses,
the
idea
that
we
actually
had
structured
the
idea
that
we
have
practices
now
some
of
those
practices
made
no
sense
by
the
way.
Okay,
I
want
to
be
very
clear,
and
that
is
a
is
a
whole
other
conversation
that
I'm
sure
simon
and
I
will
go
on
for
hours
on.
B
B
There
is
some
form
of
structure
that
there
is
some
form
of
man,
simplicity,
imagibility,
simplifying
that's
what
data
lakes
are
definitely
missing,
okay,
and
so
because
of
that,
the
idea
is
that
a
lake
house
paradigm
or
this
concept
is
saying
well
take
some
of
the
great
things
that
we
learned
about
databases,
some
of
the
great
things
we
learned
from
lakehou
from
data
warehousing
and
apply
that
to
our
data
lake.
So
that
way
our
data
lakes
become
actually
useful
right.
B
It
doesn't
preclude
the
fact
that
you
may
still
have
a
raw
zone
that
you
just
dump
the
data
because
in
some
cases
that's
still
very,
very,
very
appropriate,
but
it
does
say
that
don't
think
about
the
rigor
and
structure
as
a
secondary
thought,
if
you're
going
to
decide
to
have
a
dumping
zone,
be
cognizant
of
the
fact
that
you
still
have
that
right.
Be
cognizant
of
the
fact
that
you
better
do
something
with
that
data
soon,
otherwise,
you're
just
storing
lots
of
it
and
it's
not
very
useful.
So
hopefully
that
helps
and
simon.
A
Yeah
yeah,
it's
it's
a
similar
kind
of
I
mean
in
a
nutshell,
it's
taking
the
benefits
of
a
lake
and
its
benefit
of
the
warehouse
and
being
building
a
single
solution.
That
has
the
benefits
of
both
and
it
is
exactly
kind
of
those
things
are
listening.
Why
do
you
build
a
warehouse
because
you're
trying
to
do
financial
reporting
and
you
need
rigor?
You
need
to
be
able
to
trust
the
data
you
need
auditability,
you
need
governance,
the.
Why
is
that
missing
records?
A
Oh
someone
actually
made
an
update
transaction
and
they
removed
a
load
of
records
all
of
that
kind
of
stuff
that
is
slow
right.
So
that's
the
kind
of
why
people
built
warehouses,
they
love
warehouses
and
then
they
went.
You
know
what
we
can't
react
quickly
enough.
So
there's
kind
of
a
big
knee-jerk
reaction
away
into
lakes
and
the
flexibility
of
lakes
and
people
fell
in
loving
lakes
and
all
that
stuff
and,
as
you
said,
they
forgot
about
all
that
good
stuff
and
so
there's
now
a
reaction
against
like
so
people
going
they're
not
controlled
enough.
A
They
don't
have
enough
rigor.
They
don't
have
that
discipline
and
auditing
and
all
that
kind
of
stuff,
and
it's
it's
essentially
saying
can't.
We
just
build
a
platform
that
has
the
good
stuff
about
lakes,
the
flexibility,
the
scalability,
the
ability
to
throw
different
types
of
data
and
react
quickly,
but
per
their
layer
over
the
top
that
allows
for
transaction
history,
for
accountability,
for
security,
for
being
able
to
roll
back.
Something,
like
you
know,
the
ability
to.
Oh,
I
didn't
mean
to
hit
that
delete
query
and
all
your
parkour
files
are
gone
and
you're
like.
A
Lake
house,
daily
breaks
are
kind
of
championing
I've
been
using
that
term.
Far
too
much
is
kind
of
that.
Essentially
doing
it's,
it's
bringing
the
warehouse
functionality
into
the
lake.
It's
saying
we
want
to
improve
our
technology
stack
in
the
lake
to
be
able
to
bring
in
those
good
warehousing
features.
Now,
people
in
the
warehousing
world
they're
trying
to
do
the
same
to
bring
the
good
leki
sparky
bits
into
database
systems,
you're,
seeing
more
database
systems
adopt
some
kind
of
scalable
sparky
bits.
So
there's
lot.
A
Basically,
everyone
is
kind
of
heading
in
this
direction,
currently
to
say
they're,
both
good
ideas.
Why
not?
Both
and
lake
house
is
the
one
with
champion
here
in
terms
of
spark
in
terms
of
daily
bricks,
in
terms
of
doing
it
in
the
lake
and
making
lakes
able
to
have
governed
auditable
data
models
in
it.
B
A
C
B
Great
question,
so
one
thing
to
denote
real
quick
is
that
when
it
comes
to
the
lambda
architecture,
there's
two
facets
to
it.
There
is
the
one
that's
very,
very,
very
transactional,
based
and
that's
very
one.
That's
more
olap
dwbi
based
okay,
in
terms
of
like
the
traditional
way
of
saying
things.
The
reason
I
call
this
out
is
because
when
we
talk
about
the
delta
medallion
architecture,
it's
really
meant
for
data
warehousing
like
queries
or
olap,
be
I'd
like
queries,
it's
not
meant
as
a
replacement
for
lambda
architecture
for
highly
transactional
systems.
B
When
we
talk
about
that,
the
the
bronze
silver
gold
concepts
of
the
delta
medallion
architecture,
we're
specifically
calling
it
out,
for
you
know:
bi
olap
type,
queries,
okay,
because
this
is
a
don't
forget.
Delta
lake
is
not
a
transactional
store,
I
mean
it.
Can
it
obviously
does
transactions,
but
the
transaction
stores
is
very
much
designed
for
all
lap
queries
they're,
not
designed
for
high
throughput
high
concurrency
transactional
queries.
You
can
try
it
and
we've
seen
a
modicum
of
success
with
it,
but
we
certainly
would
not
suggest
it
right.
B
I
mean
this
is
one
of
those
cases
where
no,
it's
very
much
designed
for
the
olap
style
queries
right,
so
so
in
the
case
of
lambda
architecture
for
all
app
style
queries.
Yes,
we
feel
that
because
you
can
combine
streaming
and
batch
and
do
all
these
things
together,
that
the
medallion
architecture
is
simpler
to
maintain
simpler,
to
run
some
prefer
everything,
and
so
hence
the
reason
so
from
that
standpoint.
Yes,
we
do
think
so,
but
no
again
it's
for
olap
not
for
high
concurrency
and
transactional.
A
Lambda
quite
a
lot
right
because
it's
coming
it
it's
another
one
of
those
things
right.
It's
it's
a
workaround
around
technological
limitations.
People
are
constantly
banging.
My
report
needs
to
be
real
time
and
you're
like
does
it,
and
then
I
need
to
have
the
thing.
That's
constantly
ticking
away
on
my
wall
and
you're
like,
but
do
we
have
to
recalculate
everything?
A
And
you
get
to
that
point?
The
whole
lambda
architecture
point
is
saying:
I'm
going
to
recalculate
all
of
my
data
every
hour,
every
kind
of
day,
whatever
it
happens
to
be,
and
just
top
up
with
the
stream
and
that's
complex
right.
The
complexity,
we're
built
into
lambda
in
terms
of
managing
those
two
entirely
different
paths
is
painful,
that's
not
to
say
delta.
I've
been
doing
a
lot
with
delta,
streaming
and
kind
of
all
that
kind
of
stuff.
A
Over
the
past
two
weeks
and
I've
had
some
I've
had
some
wacky
things
that
I
could
do,
but
if
we
get
onto
that
and
it
does
simplify
stuff
right
so
saying,
I've
just
got
one
path,
I'm
just
gonna
run
it
and
I
can
either
leave
it
running
in
its
streams
or
I
can
just
run
it
occasionally
with
a
trigger
once
mechanism
and
it
batches.
I
can
do
a
bit
of
both
and
having
that
simplicity
of
going
through
a
single
architecture,
it's
a
lot
simpler
doing
the
lander
architecture.
A
Is
it
as
simple
as
having
a
batch
architecture
that
runs
as
often
as
I
actually
needed
to
do
not
just
streaming
for
the
sake
of
streaming?
No
but
yeah.
Certainly,
I
no
longer
build
lambda
architectures,
just
because
most
of
the
time
I
can
achieve
it
by
having
one
single
path.
So
for
me,
certainly
in
terms
of
that,
as
you
said,
the
olaf
pattern
that
we
used
to
have
to
build
to
enable
streaming
when
we're
dealing
with
large
data
architectures.
I
don't
have
to
do
that
anymore,
which
is
nice.
C
B
B
No
problem
so
great
question:
does
delta
lake
support
merge
on
reads?
We
actually
will
have
a
session.
I
don't
know
if
it's
going
to
be
scheduled
for
november
or
december
right
now,
but
that
actually
is
exactly
what
we're
going
to
be
doing
so
just
to
provide
some
context
from
some
folks
in
terms
of
building
transactional
stores.
They
also
you
instead
of
using
delta
lake
with
spark
they
use,
hoodie
or
iceberg
also
great
systems.
Okay,
obviously
we're
more
for
delta
lake
ourselves,
but
we're
not
gonna
knock
anybody
else
that
doesn't
make
any
sense.
B
Oh
one
of
the
really
cool
components
of
hootie
is
actually
the
concept
of
birch
on
reed.
So
in
terms
of
when
you,
in
terms
of
that
view,
that
ability
to
read
that
information-
and
so
in
fact,
like
I
said
we
have
a
session
that'll-
be
coming
up,
it's
not
on
the
official
docket
yet,
but
because
we
still
have
to
finish
the
slides
minor
technicality
so
once
we
get
that
done,
we'll
put
it
on
the
official
docket
and
we'll
actually
have
a
session
that
actually
addresses
this
exact
same
situation.
B
B
I
do
actually
yes
cool,
perfect
sure.
Let
me
start
that.
Can
I
read
10
years
later
data
using
delta
lake?
So
in
other
words,
can
I
read
data?
That's
like
10
years
later
on
it.
So
I
can
answer
I'll
start
off
and
same
idea,
simon.
You
chime
in
with
your
your
context,
but
the
the
kick
quick
thing
is
that
underneath
the
covers
of
delta
lake,
it's
parquet
files.
B
Okay,
that's
the
thing
to
remember:
it's
still:
parquet
files.
There
is
a
transaction
log
that
allows
us
to
track
which
version
of
the
files
and
how
you
read
that
transaction
log
is
completely
open
source.
Okay,
so
we've
documented
we've
commented
the
code.
So
even
if
you
don't
read
the
documents,
you
could
literally
comment
the
code,
but
for
all
intents
and
purposes
what's
happening
is
that
because
you
can
have
multiple
versions?
B
Sorry
multiple
parquet
files,
because
we
have
mvcc
multiple
version,
concurrency
control,
okay,
to
simplify
how
we
save
files.
When
you
do
inserts
updates
and
deletes
and
everything
else,
you
can
actually
have
different
versions
of
the
table.
What's
great
about
that
concept.
Is
that-
and
we
have
this
concept
called
time
travel
that
allows
us
to
go
back
in
time
and
I'm
sure
simon's
got
wonderful
stories
about
oops
and
time
how
time
travel
has
saved
everything?
B
Okay,
but
nevertheless
the
context
is
that
because
there's
different
versions
of
the
files
available,
even
if
you
weren't
to
read
the
transaction
log
directly,
you
could
all
you'd
actually
have
to
do-
is
run
one
api
that
would
generate
a
manifest
file.
The
manifest
file,
basically
is
just
a
file
that
says
here
are
all
the
parquet
files
that
you
need
to
go
read.
B
If,
then,
you
can
read
that
manifest
file
grab
all
the
parquet
files
that
are
listed
inside
there
bam.
Now,
you
can
read
know
exactly
which
version
of
the
files
you're
supposed
to
be
reading,
and
then
you
can
read
the
table
anyways,
okay,
and
by
the
way,
it's
a
huge
advantage
that
we
actually
are
using
this
manifest
for
the
transaction
log
itself.
The
reason
why
is
because
I'm
listing
lots
of
files
from
cloud
object
stores
is
actually
really
really
slow,
exceedingly
slow?
B
Well,
we're
not
making
you
list
that
now
what
we're
doing
is
simply
reading
the
transaction
log.
It
has
a
list
of
files
inside
there.
So
now
your
spark
job
or
for
that
matter,
any
job
that
wants
to
read
the
parquet
files
knows
exactly
which
files
to
to
to
query
as
opposed
to
actually
explicitly
listing
every
single
file
and
trying
to
find
out
which
file
you
want
to
use,
and
so
at
least
that's
so
in
other
words.
Yes,
this
is
a
long-winded
answer
in
terms
of
giving
a
little
bit
the
internals.
B
A
Well,
what
one
of
them
so
when,
when
the
whole
time
for
querying
thing
came
out,
I
had
a
whole
slew
of
people
go
awesome.
I've
been
building
crazy,
slowly,
changing
dimension
things
for
so
long
right.
I
don't
need
to
do
that
anymore.
I
can
just
use
temporal
queries
and
it's
like
you
can,
if
you
really
like
storing
a
hell
of
a
lot
of
redundant
data
and
that's
the
thing
to
be
aware
of
right
when
you're
talking
about
I've
got
a
parquet
file,
a
4k
file
is
immutable
right.
You
can't
change
it.
A
So
the
whole
mechanism
that
the
guys
have
built
is,
if
I
want
to
update
one
record
in
that
I
make
an
entire
copy
of
that
whole
parquet
file
with
the
change
record
and
I
copy
over
a
load
of
extra
records
from
that,
and
so
my
history
is
all
these
different
snapshots
of
that
whole
data
and
I
might
be
just
doing
tiny
little
changes
and
I'm
keeping
a
whole
sense
of
each
file
that
changed
in
that
set
and
if
you're
trying
to
say
I
want
to
keep
10
years
worth
of
history
and
you're.
Changing
that
data.
A
A
lot
you're
going
to
have
masses
and
masses
of
redundant
data,
so
it's
not
going
to
slow
things
down.
As
danny
said,
it's
just
going
to
query
only
the
files
it
needs,
but
you're
going
to
have
a
huge,
huge
volume
of
data
that
hasn't
changed
between
various
different
updates,
but
you're,
just
keeping
for
that
temporal
snapshot.
A
So
we
tend
to
think
about.
We
use
temporal
queries
as
kind
of
like
almost
like
a
transactional
backup
right,
so
we
have
like
seven
days
30
days,
something
like
that
rolling
copy
of
any
time.
I
change
it.
I
can
go
back
and
fix
it.
It's
my
safety
net.
If
I
happen
to
delete
everything,
I
can
go,
let's
undo
that,
but
it's
not
my
historical
record.
It's
not
my.
This
record
was
active
from
this
day.
To
this
day.
It's
not
slowly
changing
you
know,
so
we
tend
to.
A
A
B
B
A
That
that's
the
only
thing!
Obviously
you
can
keep
history
as
long
as
you
want
to
keep
history
as
long
as
you
want
to
pay
for
the
redundant
storage
that
you're
keeping
in
that
history,
you
can
keep
her
for
as
long
as
you
want
right,
but
just
just
don't
use
it
for
slowly
changing
just
because
it's
really
expensive.
B
Oh
absolutely,
in
fact,
we
actually
have
a
call
out
and
there's
a
summit
session
from
2020..
Sorry
does
sparking
ai
summit
2020
north
america,
man,
I'm
forgetting
the
name
of
this
session,
but
something
like
does
delta
lake
disaster
recovery,
best
practices.
I
think
that
was
called,
but
it's
with
ite
vice
and
inter
he
was
one
of
the
speakers
of
the
session,
I'm
forgetting
the
name
of
the
other
guy.
My
apologies
to
that
guy.
B
It's
not
done
on
purpose,
but
the
reason
why
I'm
calling
that
out
is
because
they
actually
call
out
exactly
what
you
just
said
simon.
They
call
out
that
like
in
like
there
is
that
concept
of
type
four
slash
historical
table.
Contents,
historical
snapshots
right.
So
if
you
want
historical
snapshots,
yeah
create
a
historical
snapshot,
then,
like
literally
just
go,
create
it
and
at
that
point
in
time,
and
then
then,
and
and
so
then,
what
will
happen
is
that
you
have,
let's
just
say,
a
monthly
fiduciary
snapshot
right.
B
So
yeah
take
a
snapshot
of
it
once
you
copy
the
data
over
then
basically
do
a
vacuum.
That
means
it'll
remove
all
of
the
files
except
for
that
snapshot.
So
then,
now
you're
taking
a
lot
less
space.
You
can
basically
put
it
on
more
cold
store
or
you
know
an
amazon
parlance
glacier.
You
know
systems,
so
whenever
you
need
to
get
access
to
it,
you
can.
But
it's
that
way
it's
really
cheap
and
the
one
thing
I
would
add
to
that
is
actually
it's
a
small
call
out
to
databricks.
B
We
actually
just
recently
introduced
this
concept
of
shallow
clones
and
deep
clones
of
your
tables.
So
basically
this
exact
example.
That's
exactly
what
you
do.
You
do
a
deep
clone
once
it's
cloned
you've
got
the
all
of
it
all
the
metadata
everything
vacuum
it
up
boom.
Now
you've
got
a
copy.
You
store
that
in
cold
storage
or
a
colder
storage
and
you're
good
to
go
so
exactly
to
what
simon's
calling
out
you
definitely
don't
want
to
be
holding
like
10
years
of
transactions
in
there.
That
would
be
a
little
bit
more
on
the
painful
side.
A
A
B
Yeah
we
we've
been
meaning
to,
and
it's
probably
on
me
by
the
way
we've
been
meaning
to
write
some
new
blogs
to
actually
explain
the
scenarios
because
we're
like
oh
crap,
we
forgot
to
do
that,
didn't
what
yep
we
did
crap
yeah.
So
it's
it's
definitely
on
our
dock
of
things
to
realize.
There's
actually
a
lot
of
really
good
scenarios
and
we
just
didn't
bother
explaining
why
you
need
to
do
it,
but
it
was
obviously
generated
from
customer
interest.
A
A
It
won't
do
a
shallow
clone,
merge
into
that,
and
it's
funny
exactly.
We
just
keep
surprising
ourselves
by
finding
more
things
like
that
of
going.
Oh
actually,
yeah,
just
just
clone
it.
Why
didn't
we
do
that,
but
yeah
I'd
say:
can
anyone
who
hasn't
really
looked
into
it?
Shallow
cloning,
specifically
just
super
useful
of
saying
I
want
something
that
looks
like
the
table.
I'm
not
copying
the
data
over
until
I
make
any
changes,
yeah
just
crazy.
B
A
Yeah
yeah
absolutely
I
mean
so
again
modern
spark
as
long
as
you're
using
data
frames.
As
long
as
you
are
using
data
frames,
there
is
no
performance
difference
whatsoever
between
pi
sparks
scholar,
r
sequel.
It
all
goes
to
the
catalyst
engine
currently
currently
until
photonic
weirdness
comes
out,
and
so
essentially
as
long
as
you're
doing
data
frame
transformations,
there's
zero
difference.
A
If
you
come
to
the
point
when
you're,
actually,
if
you're
bringing
libraries
in-
and
you
grab
like
any
old
python
library,
any
old,
our
library
and
it's
not
a
pie,
spark
or
sparkle
library,
then
that
might
have
performance
differences
if
you're
writing
a
udf.
So
you're
writing
your
own
custom
function
and
you're.
Just
writing
straight
on
python.
Then
the
differences
of
python
and
r,
don't
compile
down
to
java,
so
spark
entirely
runs
inside
a
thing
called
jvm
and
java
virtual
machine.
A
So
when
you
try
and
run
this
little
bit
of
python,
if
you're
writing
pi
spark
on
top
of
a
data
frame
that
just
runs,
it
gets
turned
into
java
bike
and
everyone's
happy.
If
you've
got
a
little
bit
of
custom
bit
of
python,
that
has
to
call
out,
for
that
has
to
go
outside
the
jvm
and
there's
an
interrupt
and
you
can
get
it
faster
with
things
like
vectorized
and
panda
pandas
udfs,
but
it's
never
as
fast
as
doing
it
in
a
data
frame.
B
Cool
and
just
to
add
to
simon's
callout
exactly
what
he
said.
The
one
awesome
improvement
that
we
did
do
is
specifically
when
you
do
create
udfs,
at
least
in
python,
make
sure
it's
a
vectorized
udf.
Okay,
it's
certainly
the
we
did
a
bunch
of
improvements
spark
2.3.
We
did
a
bunch
of
improvements
in
spark
3.0,
but
the
call
out,
basically,
is
that
if
you
build
a
vectorized,
udf
or
sorry
pandas
udf,
vectorize,
the
old
name
pandas,
the
current
new.
B
So
if
you
build
a
pandas
udf,
the
one
that's
happening
is
that
basically
it's
able
to
still
take
advantage
of
the
of
the
distributed
nature
of
the
spark
engine,
and
so
then
you
can
distribute
everything
anyway.
So
you'll
often
see
us
use
pandas
udfs
to
distribute
even
what
is
conceivably
a
single
node.
B
B
C
B
Oh
pipeline
logs,
okay,
that's
a
little
bit
more
detailed
okay!
So
now,
in
the
case
of
pipeline
logs,
are
I'm
gonna?
I'm
gonna
actually
make
it
a
two-part
question
just
because
there's
two
different
ways
of
interpreting
that
statement.
Okay,
you
can
talk
about
pipeline
logs
in
terms
of
spark
pipeline
logs
like
as
in
you're,
doing
etl
processing,
or
you
can
also
think
about
it
in
terms
of
machine
learning
pipeline
logs,
okay.
B
So
I'm
going
to
do
the
easier
one
first,
which
is
machine
learning,
okay.
So
in
the
case
of
machine
learning
I
would
just
simply
use
mlflow
okay.
The
reason
why
I
use
mflo
is
because
I
can
just
basically
when
you
work
with,
especially
in
the
database
echo
system,
but
this
is
even
part.
A
sizeable
chunk
of
this
stuff
is
actually
even
available
within
oss
ml
flow.
By
the
way,
I
wouldn't
be
very
clear
about
that.
Automatically.
B
The
tracking
of
the
metrics
of
the
parameters
is
automatic,
actually
in
mflow,
so
it'll
just
track
all
of
those
different
things
as
part
of
your
pipeline.
So
when
you
build
an
ml
pipeline,
like
you,
do
td
idf
you're
doing
some
linear
regression
or
logistic
regression
whatever
it
is
that
you're
doing
right,
a
lot
of
this
stuff
actually
is
automatically
recorded,
so
you
don't
really
need
to
from
a
machine
learning
perspective
like
log
as
much
stuff,
because
it's
all
the
parameters
and
information
you
care
about
is
automatically
logged
into
ml
flow,
anyways.
B
B
We
talked
about
this
in
the
past,
but
the
context
basically
is
that
there
are
mechanisms
that
will
simplify
within
the
databricks
ecosystem
and
how
you
can
automatically
log
and
have
lineage
when
it
comes
to
processing
your
data
in
the
case
of
you're,
saying
no,
no,
I'm
just
looking
for
open
source
spark
pipelines.
Okay,
that's
fair!
So
in
terms
of
that
etl
process,
what
you
really
want
to
do
is
you
want
to
in?
It's
really
depend
on
which
cloud
infrastructure
you're
actually
working
with,
for
example
like
if
you're
on
azure.
B
And
if
you
have
other
systems
that
you're,
also
working
with
within
azure
or
in
aws
respectively,
then
there's
the
advantage
of
that.
The
advantage
of
writing
it
out
yourself
is
that
actually,
you
can
put
a
level
of
detail
and
a
debugging
level
that
you
would
not
necessarily
have
had
by
using
some
other
system,
and
so
it
really
depends
on
what
is
it
you're
trying
to
achieve?
And
so
I
I
apologize
for
being
a
little
bit
more
vague.
But
as
you
can
probably
guess,
we
can
really
rat
hole
on
this
one.
B
A
I
mean
yeah,
I
mean
I
can
talk
about
what
we
do.
You
know.
So,
from
our
framework
point
of
view,
we
tend
to
classify,
it
is
we've
got
kind
of,
I
guess,
application
level,
logging
and
telemetry
level
logging.
You
know
so
we're
in
azure,
so
I've
got
databricks.
You
can
just
poke
it
and
say
kick
out
logs
to
log
analytics.
A
We
have
like
a
little
custom
function,
wrapper
that
we
kind
of
poke
and
say
add
some
more
contextual
logs,
add
a
little
more
parameters,
take
the
output
of
spark
streaming
and
get
the
resonant
progress
and
put
that
out
to
log
analytics
now.
So
there's
a
few
things
where,
if
it's
fairly
verbose,
we'll
also
push
that
to
log
analytics,
but
then
a
lot
of
what
we're
doing
a
lot
of
the
processing
we're
driving
based
on
metadata.
So
we
have
like
a
little
tiny,
sql
db,
which
is
just
what
are
my
process?
A
A
This
is
just
the
general
state
of
stuff
we'll
throw
out
to
log
analytics,
mainly
because
and
actually
said
anything
we've
got
we're
in
azure
we've
got
data
factory,
we've
got
sql
db,
we've
got
loads
of
other
things
that
are
also
kicking
their
logs
out
their
central
place
and
you
don't
end
up
with
like
well.
No
one
likes
siloed
data
right
and
logs
isn't
the
type
of
data,
so
just
put
it
all
in
one
place,
and
then
you
go
diving
in
so.
For
us
that
makes
sense
for
us.
B
Cool
rock
on
dude
all
right,
there's
a
we
only
have
about
11
minutes
left
or
actually,
probably
sorry.
We
need
to
make
sure
that
we
have
time
for
karen
to
wrap
things
up
so
more
like
like
six
minutes
left
so
okay,
let
me
stream
through
some
real
quick
questions
that
I
just
noticed
on.
Linkedin
live
so
at
least
I
can
answer
some
as
many
of
them
as
fast
as
I
can
before
we
get
into
some
of
the
more
detailed
ones.
All
right
does.
Delta
lake
use
mvcc
as
a
concurrency
algorithm.
B
It
does
that's
the
quick
question
answer
on
that
one.
It
does
because
the
idea
is
that
we
are
copying
the
files
down
and
we're
replicating
it.
So
in
some
ways,
just
like
simon
called
out
earlier,
it's
it's
messier
from
the
standpoint
that
it
uses
more
storage,
but
in
return
for
that,
you
actually
have
you
can
replay
and
recreate
the
entire
table
from
scratch,
which
is
really
really
nice.
B
Okay,
let's
see,
can
we
have
an
advanced
version
of
notebooks
and
workspace
for
developers
for
writing
code
and
we'd
like
to
have
ide
like
features
yes
with
databricks.
You
can
actually
use
databricks
connect
that
allows
your
ide
to
talk
to
your
of
choice
like
visual
studio
code
or,
yes,
I
actually
said
visual
studio
code
or
intellij
or
whatever
else,
to
go
ahead
and
actually
talk
to
databricks
as
well.
So
no
problem
at
all.
B
How
is
versioning
happening
in
delta
lake,
just
as
we
called
out
the
mvcc
automatically
as
part
of
the
transaction
log
automatically
takes
that
care
of
that.
For
you,
let's
see,
I
think
that
was
oh
here
we
go
final,
one,
that's
a
fast
question!
Sorry!
Do
we
need
to
optimize
and
vacuum
delta
format,
tables
for
storage,
optimization
on
a
regular
basis?
If
we
don't
bother
about
versioning,
is
there
no
default
spark
variation
setup?
B
The
defaults
for
optimize
and
vacuum
are
basically
seven
days
for
the
data
30
days
for
the
log
okay,
the
idea
is
that
when
you
instantiate
a
delta
table,
the
whole
purpose
is
that
you
want
to
be
able
to
go
ahead
and
do
changes
to
the
table.
So
by
the
very
definition
of
it,
you
will
go
ahead
and
have
a
version
because
you're
inserting
new
data
into
it
or
you're,
updating
or
you're.
B
Deleting
or
merging
or
whatever
else,
if
you're
gonna
go
with
the
defaults,
usually
that's
more
than
good
enough,
because
when
you
run
like
you
know,
compaction
or
when
you
run
a
vacuum
or
whatever
else,
it'll
actually
remove
the
data.
So
there's
only
like
seven
days
worth
of
data,
for
example
stored
inside
there
and
so
that'll
keep
the
the
table
relatively
small.
In
comparison,
that
means
there's
only
seven
days
worth
of
data
quote-unquote.
That's
that
has
possible
not
the
duplication,
but
some
days
of
versions
of
data.
I
should
say.
B
A
You
need
to
run
and
optimize
for
it
to
optimize
unless
you've
got
auto
optimized,
which
is
slightly
different,
and
you
need
to
run
a
vacuum
for
it
to
use
those
seven
days.
It's
just
there
to
default.
What's
going
to
happen,
if
you
just
throw
those
commands
with
no
other
settings,
let's
go
it's
going
to
go
now.
B
A
It
depends
every
time
yeah
baby
well,
so
you've
got
basically
three
different
parts
of
people
take
and
it
depends
on
so
so
often
we
go
here's
the
most
performant
route
and
people
go,
but
it's
not
pretty
like
so
option,
one
that
we
use
most
of
the
times
hashing,
so
actually
not
surrogate
keying
at
all.
Just
throwing
a
hash
over
a
composite
of
whatever
makes
your
primary
key.
Whatever
makes
your
change
key,
throwing
that
in
there
and
then
actually
kind
of
putting
the
pain
on
the
person
reading
it
now.
A
So
if
you
try
and
do
like
a
window
function
with
row
number
then
you're
actually
forcing
it
to
sort
your
data
and
actually
just
do
all
that
and
that's
never
going
to
be
particularly
fast,
whereas
monotonically
increasing
id
will
give
you
a
an
increasing
id
inside
each
of
your
partitions,
which
is
much
much
more
efficient,
but
not
never.
Gonna,
look
quite
as
pretty
as
your
existing
things,
but
it
means
you
end
up
doing
the
what's.
A
My
maximum
run
currently
in
my
dimension
table
add
in
add
that
to
my
monotonically
increasing
id
I'm
getting
from
my
new
ones
and
then
put
that
in
that
means
you
can't
quite
do
merges
the
same
way.
So
it
kind
of
depends
on
how
quick
you
want
your
etl
to
be.
If
you're
desperate,
to
have
survey
keys,
then
there
are
patterns
you
can
do
around
that
kind
of
manual
generation
which
feels
super
old-school
right.
It
feels
like
we're
going
back
in
time
before
databases
have
identity
columns,
because
we
kind
of
are.
B
Perfect,
so
let
me
add
to
simon's
point:
we're
gonna
need
to
share
this
by
the
way.
So
the
reason
I'm
saying
simon's
point
is
because
we
actually
had
a
tech
talk
on
august
25th
on
this
exact
topic,
introduction
to
sergey
key
generation
for
your
lake
house
and
again
this
was
not
a
question
that
was
posed
by
anybody
that
I
know
so.
Thank
you
very
much
for
posting
the
question
on
youtube.
Live.
B
We
actually
call
that
out
and
if
it's
okay
with
you,
the
reason
I'm
showing
is
because
I
have
a
ton
of
stargate
sg-1
references
inside
there.
So
that
tells
you
how
much
of
a
geek
I
am
so
this
entire
circuit
key
concept
because
it
said
sj
circuit
key
and
then
I
thought
of
sg
and
then,
as
soon
as
I
thought,
sg,
I'm
thinking,
hey,
stargate
and
then
now
I
had
a
lame
ass
reason
to
go
ahead
and
go
somehow
include
samantha
carter,
because
she's
awesome
of
this
g1
into
the
concept
of
circuit
keys.
B
So
there
is
a
we'll
share
that
particular
notebook
out
to
everybody
before
we
close
off.
Let
me
go
ahead
and
do
two
questions.
I
just
realized
I
forgot
to
ask
from
zoom
one
was
that
does
delta
lake
hammond
help
with
managing
the
schema?
Yes,
delta
lake
itself
does
schema
evolution
and
scheme
enforcement.
So
there
is
this
concept
of
scheme.
B
The
scheme
is
actually
stored
within
the
transaction
log
itself,
not
in
the
footer
fire
of
the
parquet
files,
so
that
will
simplify
things
number
one
and
so
for
history
and
simon
I'd
love
you
to
chime
in
on
this
one.
As
the
last
part,
do
you
still
recommend
implementing
an
scd
design
for
history
bam.
A
A
If
you
need
history,
absolutely
again,
as
we
said
type
4
is,
is
my
preferred
just
because
it
means
people
can
quit,
because
summerville
keys
can
be
a
little
bit
awkward
to
generate.
Then
separating
out
between
this
is
my
current
table.
I
can
just
join
straight
to
without
having
to
worry
about
tamper
context
or
go
into
my
history
table,
and
at
that
point
I
can
add
the
complexity
of
choosing
the
right
version
of
my
time
and
do
a
range
skew,
join
and
pass
in
the
range
cue
joint
hint
which
speed
that
stuff
up.
A
Then
yeah
you've
got
that
choice
and
making
it
simple
by
separating
out
the
current
and
the
historical
definitely
makes
things
faster
and
makes
etl
a
little
bit
simpler,
and
especially
if
you're
doing
things
like
you
can
do
delta
streaming.
Now,
if
you
want
to
go
crazy,
where
you
update
your
current
table
and
then
stream
out
those
changes
to
your
historical
table,
so
you
can
kind
of,
like
asynchronously,
offset
the
creation
of
your
history
from
your
insert.
If
you
want
to
make
things,
fancy.
B
B
C
Oh
yeah
thanks
denny
thanks
simon
and
thanks
everyone
for
all
of
your
questions.
Again:
apologies
for
not
getting
to
the
to
all
of
them,
but
like
then
he
said
we
might
have
to
plan
a
third
session.
So
I
guess
guess
we'll
see
what
scheduling
looks
like
maybe,
but
thanks
everyone
for
joining
us
and
I
just
dropped
the
the
link
again
to
our
data
plus
ai
online
meetup
group
in
linkedin
and
youtube
live
and
in
the
zoom
chat.
So
that's
the
best
way
to
to
get
notified
of
our
upcoming
sessions.