►
Description
Considering shifting gears into Spark Data Engineering? We have another fun session with Simon Whiteley (@mrsiwhiteley) and Denny Lee (@dennylee) as they answer your questions from their last two tech talks and continue to chat through their meandering journeys from SQL Server & BI to Apache Spark, Delta Lake, and the emerging Data Lakehouse approach. Be prepared for another geeky, trans-Atlantic event from two data nerds.
Link to watch round 1: SQL Server and BI to Delta Lake and Lakehouses: https://youtu.be/ls7-LKwKFYA
Link to watch round 2: Electric Boogaloo: https://youtu.be/OoQ21kQoJ84 Databricks is proud to announce that Gartner has named us a Leader in both the 2021 Magic Quadrant for Cloud Database Management Systems and the 2021 Magic Quadrant for Data Science and Machine Learning Platforms. Download the reports here. https://databricks.com/databricks-named-leader-by-gartner
B
B
Okay.
So
for
those
of
you
who
are
familiar
with
kimball
or
iman,
there,
you
go
so
so
we
had
him.
We
had
susan
o'connell
and
donald
farmer
for
those
of
you
that
are
in
the
microsoft
sql
server
space.
Then
you
are
pretty
familiar
with
those
names
as
well,
so
there
you
go
so,
but
nevertheless
the
the
reason
I
started
that
wasn't
just
to
go
ahead
and
chime
in
about
the
data
brew,
vidcast
series.
B
It's
also
to
go
ahead
and
call
out
the
fact
that
that's
exactly
where
both
simon
and
I
are
coming
from
right.
We
came
from
this
context
of
sql
server,
relational
database,
lan
and
now
we're
you
know
flow
through
to
sql
bi,
big
data
spark
and
now
into
the
realm
of
delta
lake
data
lakes,
lake
houses
right
and
it's
been
a
fun
fun
journey
to
put
it
lightly.
So
my
just
in
case,
you
don't
know
who
I
am
my
name
is
danny
lee.
B
I'm
a
developer
advocate
I've
been
working
within
the
space
for
a
long
time
and
yeah
we're
prepared
for
your
questions,
technical
questions,
okay,
so
please,
chime
in
in
any
of
the
forums,
whether
it's
linkedin,
zoom
or
through
youtube
simon.
Why
don't
you
go
ahead
and
do
a
quick
introduction
for
yourself
for
those
that
may
not
know
who
you
are
sure.
C
Thank
you
see
ya,
hello,
I'm
simon
hi.
I
am
from
the
uk
doing
various
things
around
daniel
engineering
and
all
that
stuff,
as
danny
said,
came
originally
from
heavily
microsoft,
bi
land,
and
these
days
I
spend
most
of
my
time
building
lakes
and
spot
and
teaching
people
how
to
do
framework,
driven
engineering
and
a
lot
at
the
moment,
especially
this
past
two
weeks,
teaching
sql
people
how
to
work
in
a
databricks
environment
which
has
been
interesting
going.
C
B
Okay,
cool
tonight,
so
self
promo.
No,
no!
No!
No!
It's
perfectly
fine
for
self
promo.
We're
perfectly
fine
doing
that
and
by
the
way,
just
to
provide
some
context.
We
will
actually
be
talking
a
little
bit
about
the
date
and
ani
summit
partially
as
a
promotion
to
data
item,
but
also
because
both
simon
and
myself
have
some
pretty
cool
sessions,
and
so
we're
probably
just
going
to
provide
a
little
bit
of
quick
context
as
a
way
to
encourage
you
to
go,
attend
our
sessions
because
guess
what
we've
got
some
cool
geeky
stuff.
B
B
C
B
C
Exactly
with
the
whistle
there
we
go,
I
do
have
to
apologize.
There
might
be
some
explosions
coming
in
the
background,
because
today
is
november,
the
5th
which
is
fireworks
day
in
england.
When
we
celebrate
blowing
up
someone
who
tried
to
blow
up
parliament
because
that's
what
we
do
so
you
know
you
might
hear
some
things
just
ignore
it.
It's
fine!
No
one's
blowing
up,
it's
okay,
yeah!
Okay,
I
mean
so
the
first
one
is
so
did
we
do
that?
C
C
I
mean,
and
it's
it's
one
of
the
biggest
things
that
we
get.
I'm
kind
of
the
you're
building
a
lake
you've
got
your
various
structures.
You've
got
your
bronze
silver
gold
raw
basin,
rich.
Whatever
you
want
to
call
your
three
tiers
of
I'm
building
a
lake
and
I'm
segmenting
data,
and
almost
every
time
I
speak
to
people
they
try
and
go.
I
need
to
pick
out
a
partition
and
I
partition
the
same
way
all
the
way
through,
and
you
just
absolutely
don't
have
to
right.
For
me.
There's
there's
kind
of
source
based
etl
partitioning.
C
C
So
you
can
do
things
like
say
yes
replace
that
partition,
you
know
and
you
can
do
merges
and
add
in
the
partition
constraints,
so
you're
just
merging
in
just
to
that
window,
and
it's
all
about
how
do
I
optimize
that
one
right
transaction?
How
do
I
get
data
into
the
table
in
the
fastest
way
possible,
but
then,
once
you've
landed
the
table
in
some
kind
of
customer
facing
analytics
style
thing,
your
gold
layer,
your
curated
layer,
whatever
you
want
to
call
it,
then
it's
all
about
who's,
going
to
query
that.
C
C
I
know
I
know
I'm
loading
in
an
hour
batch
or
a
day
batch
or
a
week
or
whatever
it
happens
to
be,
whereas
if
I'm
doing
serving
something
to
the
users,
I'm
having
to
guess
and
saying
what
kind
of
what
kind
of
search
predicates
are
people
going
to
put
in
their
where
statement
how
people
are
going
to
open
power,
bi,
tableau
and
filter
and
slice
that
data?
What
kind
of
queries
are
people
going
to
write
and
that's
really
hard
to
know
up
front?
C
You
know
so
kind
of
the
the
art
is
in
getting
it
down
getting
it
out
there
trying
to
look
at
how
people
work
with
it
and
then
going.
You
know
what
I
might
have
to
just
lift
that
whole
table
up
and
re-partition
it
differently
and
put
it
down
just
because
people
aren't
hitting
those
partition
keys
and
it's
tricky.
But
it's
all,
how
are
you
loading
data
and
then
at
some
point
you
switch
over
to
how
people
are
going
to
query
it.
B
B
Well,
the
question
was
related
to
history
and
time
travel
is
that,
do
you
still
recommend
implementing
a
slowly
changing
dimension,
type,
2
design,
and
so
for
those
of
you
just
to
provide
a
little
quick
context
and
simon
definitely
chime
in
if,
if
you
feel
that
I'm
missing
anything,
but
we
so
did
it
commonly
in
any
of
the
any
old
app
design,
not
necessarily
just
about
analysis
services
cubes,
I
mean
olap
queries
in
general
right.
The
context
was
that
you
had
a
dimension
table
and
you
had
your
fact
table
right.
B
Don't
worry,
I'm
not
going
to
go
into
factless
fact
tables
or
anything
like
that,
just
just
pure
dimension
and
pure
fact,
tables
and
things
change
over
time.
So
forsake
argument.
Demographics,
information,
like
your
address,
like
I
used
to
live
in
in
st
john's
newfoundland
and
then
over
time
I
moved
to
montreal
and
then
over
time
I
moved
to
seattle
right.
B
Anything
that
was
done
20
years
ago
was
montreal
and
then
the
rest
in
the
seattle
area
right.
The
reason
I
bring
that
up
is
that
the
people
are
going
well
yeah.
Well,
I
don't
need
to
do
that
now
with
data
links
anymore.
Do
I
right-
and
in
fact
it's
definitely
one
of
those?
Yes,
you
definitely
can
do
it
and,
yes,
you
should
do
it,
but
also
it's
an.
It
depends
question
too
right.
So,
in
other
words,
it's
not
it's
not
like.
You
can't
just
go
ahead
and
put
everything
in
the
facts.
B
You
you
absolutely
can't
so,
in
other
words,
forsake
argument.
I
go
ahead
and
put
all
the
demographic
information
into
the
facts,
and
it's
all
there
and
that's
fine
except
okay,
don't
worry,
except
exactly
the
same
problem
as
we
had
with
data
warehousing
traditionally
or
olap
cubes,
and
traditionally
your
your
the
dimensions.
Don't
change
that
often
right
at
least,
I
certainly
hope
not.
I
hope,
you're
not
moving
that,
often
right
that
would
resort
in
massively
large
dimension
tables.
B
So
then
the
return
statement
is
that
yeah,
that's
even
true
for
data
lakes,
especially
if
you
have
a
really
wide
demographics
table
like
if
you
include
all
the
content
or
facets
about
a
particular
person
as
part
of
your
machine
learning,
algorithms
right
that
you
actually
want
to
understand
various
segmentation
or
demographics
that
they
belong
to,
so
it
still
slows
into
it.
It's
still
not
fast
changing
right.
It's
still
slow,
then
changing
right,
even
if
you
include
in
all
the
other
facets
right
like.
B
Oh,
you
have
kids
right,
you're,
not
having
kids
fast
enough
to
warrant
that
as
a
fast
changing
dimension
right
so
because
you're
doing
stuff
like
it's,
because
it's
still
a
slow
change
dimension
that
table
that
dimension
table
is
still
going
to
grow
significantly
slower
than
your
fact.
And
then,
if
you
were
to
put
all
that
information
into
the
fact
table
quote-unquote,
it
would
just
massively
grow
for
no
good
reason
right
at
all
right.
So
yes,
you
absolutely
would
want
to
go
ahead
and
do
slowly
changes,
dimension,
design
and
then
there's
about
the
technique.
B
So
I'm
not
going
to
get
too
deep
into
that
and
the
reason
I'm
not
is
because
I'm
just
simply
going
to
send
you
all
a
link,
I'm
going
to
paste
it
into
linkedin
in
a
second
and
also
into
zoom
into
youtube.
The
reason
why
is
because,
in
fact,
douglas
moore
and
myself
douglas
moore
also
comes
from
a
very
relational
database
background.
That's
exactly
the
topic.
We
actually
spent
an
hour
just
on
that
topic
about
scd
type
2
by
the
way,
there's
also
a
topic
just
on
surrogate
key
generation
as
well.
B
So
so,
like
I
said,
if
you
are
to
ask
me
that
question
about
fast-changing
dimensions,
we
would
then
simon-
and
I
would
go
on
for
about
five
hours,
to
explain
where,
where
we
would
deviate
and
why
right.
But
the
reality
is
because
there's
when
it
comes
to
fast-changing
dimensions,
it
does
warrant
the
idea
of
us
going
and
actually
putting
them
into
facts.
But
then
that
goes
into
factless
fact:
tables
and
fun
stuff
like
that
right.
B
But
when
it
comes
to
slowly
changing
dimensions
at
least
there's
you
absolutely
are
warranted
for
doing
such
a
thing.
So
hopefully
that
answers
that
question.
I'm
gonna
go
ahead
and
paste
the
link
now,
just
because
I
happen
to
have
it
handy,
it's
because
it's
on
the
same
tech
talk
series
that
we're
currently
doing
right
now,
so
it
becomes
really
handy
but
yeah.
I
think
hopefully
that
answers
that
question
concerning,
should
I
do
std
type
2
in
data
lakes
and
the
answer
unequivocally
is
yes,
you
should.
B
B
C
Okay,
I'm
gonna,
I'm
gonna
have
a
minor
minor
little
little
rant.
There's
one
of
the
questions
very
specific.
It's
like
a
super
niche
question.
Okay,
all
right
we're
going
save
as
table
function.
Does
that
use
a
separate
catalog
to
the
other
hive
catalogs.
So
it's
like
real
real
drilling
down
thing.
C
Basically,
if
you've
got
a
data
frame,
you're
writing
something
in
spark
and
you
go,
I
kind
of
got
what
I
need
dot
savers
table
and
you
want
that
to
register
in
hive,
so
you
can
then
go
and
just
go
select
start
from
table
at
a
later
date.
Right.
Super
super
useful.
That,
for
me,
is
the
most
misleading
evil
function
known
to
man,
because
yeah
it's
great
now,
no,
it
doesn't
use
a
separate
hypercap
like
it
does
register
it
with
your
hive
catalog.
C
So
it's
a
standard
registry,
but
what
that
does
is
that
makes
a
separate
copy
of
your
data
in
dbfs
right
second
to
take
wherever
you're.
So,
if
you're
saying
I
want
to
take
something
from
my
lake,
my
nice
curated
secure,
managed
lake
and
I
want
to
make
a
copy
of
it
in
the
local
data.
Databrick
storage,
which
is
then
not
accessible
to
anywhere
else
in
the
data
ecosystem.
C
C
You
can't
see
it
outside
of
the
database
ecosystem,
so
your
cluster
has
to
be
spun
up
to
be
able
to
query
it
and
all
sorts
of
madness
so
save
this
table.
When
you
first
see
it
it's
like.
That's
super
useful
yeah.
I
just
want
to
save
it
as
a
table,
but
we
nearly
always
recommend
not
using
it
just
because
you're
intrinsically
too
tightly
coupling
the
hive
registration
with
your
actual
data,
and
it
nearly
always
leads
to
problems
down
the
line.
So
we
never
use
it.
I
don't
know
if
you
agree
with
that.
Danny.
B
No,
actually,
I
completely
understand
the
context
because
in
in
all
seriousness,
this
is
a
much
more
complicated
answer
than
most
people
realize.
Exactly
to
your
point
right,
because
when
it
comes
to
using
your
catalogs,
you
actually
need
to
understand
your
overall
ecosystem.
It's
not
just
about
the
data
lake.
Don't
get
me
wrong.
Data
lake
is
super
super
important.
Obviously
right
it
is
at
the
risk
of
sounding
like
terror
data,
which
is
horrible.
B
Yes,
I
said
that
it's
your
central
source
of
truth
right,
but
the
thing
is
that
even
having
a
central
source
of
truth,
the
value
is
what
you
get
out
of
it
right,
and
so
some
folks
are
going
to
be
able
to
go
ahead
and
have
more
homogeneous
design.
So
I'm
not
I'm
actually
trying
to
answer
very,
very
non-ver,
very
vendor
agnostic,
so
I'm
just
going
to
say
homogenous
versus
heterogeneous.
B
B
That
is
almost
consistently
never
going
to
happen,
okay
and
then
just
to
add
a
new
wrinkle
to
it
if
you're
multi-cloud
multi-environment,
like
in
other
words
on-prem
and
cloud
and
multi-cloud,
which
a
lot
of
enterprises
are
actually
either
doing
or
starting
to
do
right
now.
That
in
itself
is
its
own
set
of
problems
right
and
so
exactly
to
simon's
point,
then
how
you
decide
what
you
do
with
your
catalog
and
how
you
connect
to
it
whether
it's
save
a
table,
even
though
it
just
starts
with
a
save
as
table
statement
right
there.
That's
the
implication.
B
The
implication
is
where
what
how
you're
going
to
interact
with
cuddle?
That's
actually
why
there
are
vendors
that
build
their
own
catalog
services
right.
Some
of
them
are
tightly
integrated
to
their
environment.
Some
of
them
are
completely
meant
to
be
vendor
agnostic,
some
of
them.
You
have
to
maintain
yourself,
which
blows,
but
it's
actually
important
in
some
cases.
So
that's
why
the
even
though
it's
a
small
little
statement
to
say
save
a
table,
it
actually
implies.
B
So
many
different
things
that
yeah
literally
it's
obvious
that
you
and
I
can
probably
go
all
four
hours
just
on
this
statement
too,
so
so
to
try
to
cut
it
short.
So
we
can
actually
answer
other
questions.
Yeah
the
reality
is
exactly
to
simon's
point.
It
really
depends
on
your
environment
before
you
decide
something
as
simple
as
save
a
table
or
not
cool.
You
want
to
dive
into
the
next
question
or
you
want
me
to
dive
into
it.
C
B
You
can
pick
a
question:
why
not
perfect
I'm
going
to
pick
one
run
from
linkedin
linkedin
live,
there's
a
question
here
from
thomas
or
tomas.
I
hopefully
I
said
your
name
correctly:
can
you
use
delta
lake
open
source
in
aws
without
databricks
runtime?
So
that's
a
easy
answer.
That's
why
I
decided
to
pick
that
one
absolutely
go
to
delta.io
go
to
github.com
delta
dash
io.
B
You
can
absolutely
download
the
code
base
yourself.
You
can
run
it
on
aws
without
any
problem
we
have.
Actually
there
are
many
customers
or
many
in
the
community.
I
should
put
that
way.
Many
enterprises
that
actually
do
that.
There
are
some
issues
when
it
comes
to
exactly
which
version
of
spark
you're
using.
B
So
you
have
to
make
sure
you
get
the
versioning
numbers
right,
but
outside
of
that
yeah,
no,
you
absolutely
100
can
go
ahead
and
use
it
in
open
source
and-
and
it
is
an
open
source
project
you
can
go
to
github.
Take
a
look
at
the
code
base.
Give
us
prs,
give
us
issues,
because
that's
actually
what
we
want
to
have
so
yeah,
absolutely.
B
C
To
start
with
that,
one
and
I
can
find
so
many
different
places
yeah.
I
know
yes,
okay,
so
if
I
use
data
lake
or
sparkler
tableau
do
I
need
a
caching
layer
for
interactive
analysis,
analysis
to
get
enough
performance,
and
you
know
what
I'm
going
to
say.
What's
enough,
if
I
say
enough
performance,
how
much
is
enough
performance
and
it
is
a
it's
a
delicate
balance.
It's
the
consultant's
answer.
It
depends,
generally,
you
can
do
it
a
lot
you
can.
C
You
can
get
a
hell
of
a
lot
of
performance
over
large
data
sets
querying
the
spark
engine
directly.
Can
you
do
millisecond
style?
I
want
to
click
on
things
and
have
things
immediately
act?
No,
it's
a
parallel
system,
there's
always
an
overhead
of
parallelism.
So
it's
always
about
what
is
enough.
Is
it
until
people
aware
actually
I'm
creating
a
few
billion
rows
here?
If
I
put
a
slicer
and
it
takes
three
four
five
seconds
to
come
back,
you
know
what
that's
that's
cool,
because
I
was
never
able
to
get
at
this
data
before.
C
But
now
I
can-
and
that's
amazing
great
if
people
are
looking
at
it
and
going
what's
it
doing,
what
no
that's
too
slow,
then
yeah.
You
need
a
cash
anyway,
so
it
depends
on
the
the
style
of
data,
the
style
of
interactivity
and
the
expectations
of
the
users.
In
terms
of
what
that
turnaround,
speed
needs
to
be
in
terms
of
what
I
would
recommend
for
an
intermediate
caching
layer.
It's
ecosystem
dependent,
you
know,
so
I'm
tend
to
be
in
azure.
C
People
tend
to
use
power
bi
for
that,
and
you've
got
all
sorts
of
different
modeling
techniques
you
can
use.
Most
of
the
caching
layers
tend
to
have
something
these
days,
which
is
all
this
kind
of
there's
a
new
modern
approach,
which
is
actually
exactly
the
same
as
the
old
whole,
app
approach,
which
is
essentially
saying
and
keep
some
stuff
in
memory
and
some
stuff
on
your
spark
plus
there,
and
if
you
can
get
that
balance.
That's
what
I
look
for
in
a
caching
tool
right,
something
where
I
can
go.
C
If
someone
tries
to
run
a
query
and
they're
this
great,
you
know
they
try
and
summarize
things
by
product.
That's
cool
that'll,
get
it
from
the
cache
and
if
they
in
that
same
dashboard,
can
drill
down
and
say
actually
give
me
the
transaction
level.
Then
that
is
intelligent
enough
to
go
and
that
one
I'm
going
to
throw
to
spark,
because
that's
where
the
big
data
engine
is
that's
it
as
long
as
the
caching
engine
can
do
that.
So
all
of
your
tableaus,
your
drameos,
your
power
bis,
your
kind
of
various
different
analytics
semantic
layers.
C
They
all
have
things
like
that
baked
in
so
as
long
as
that
makes
sense
and
you're
building
models
in
a
smart
way.
So
you're
saying
I'm
only
going
to
put
essentially
cached
data
is
expensive.
To
put
something
in
an
in-memory
cache
is
tends
to
cost
money.
You
know
you're
paying
a
premium
for
that
stuff.
That
has
that
interactive,
fast
reaction.
C
So
if
you
can
split
out
and
say
you
know,
80
percent
of
our
queries
are
at
this
level,
we'll
put
that
in
cash
and
then
everything
else,
the
occasional
user
who
asks
the
hard
question
and
what's
the
million
row
answer,
then
they
have
to
wait
a
few
seconds
to
go
back
to
the
spark
engine.
Then
that's
the
kind
of
design
you're
looking
for
so
whatever
tool
can
do
that
in
your
ecosystem
makes
sense.
B
Perfect,
I
definitely
would
love
to
answer
answer
it
in
a
two-parter
by
the
way.
So
the
first
part
is,
is
a
small
plug.
You
know
have
to
plug
it
a
little
bit,
there's
the
photon
engine
from
databricks
that
actually,
which
we
announced
during
ignite,
actually
the
microsoft
ignite
conference,
where
it
significantly
improves
the
performance
of
your
speed
on
your
data
lakes,
okay,
so
it's
designed
specifically
for
that
it
doesn't
preclude
what
simon's,
saying,
though,
I
want
to
be
very
clear
about
that.
B
It
basically
just
means
the
part
that
what
simon's
saying
about
I
want
to
be
able
to
look
at
my
billions
of
rows,
and
I
want
that
to
come
back,
not
in
five
hours.
Okay.
Yes,
that's
what
the
photon
engine
is
designed
extremely
well
for,
for
that
purpose,
all
right
versus
there
are
going
to
be
absolutely
some
scenarios,
especially
where
you
like,
just
like
you,
said
80
of
your
queries
where,
frankly,
you
don't
need
to
have
all
the
data
you
want
to
be
you
you
need
to
make
it.
B
You
need
to
make
the
the
queries
more
efficient
right
for
that
purpose,
for
those
can
reporting
for
the
ad
hoc
reporting
for
the
that
line
of
business
things
of
that
nature.
Absolutely
then,
it
makes
a
ton
of
sense
for
you
to
go
ahead
and
build
whatever
caching
layer
and
again
it
that's
very
ecosystem
dependent.
B
The
one
thing
I
I
would
like
to
call
out
is:
actually
it's
actually
very
reminiscent
and
sorry
simon
for
me
pulling
up
this
old
one
again
of
the
yahoo
24
terabyte
cube
right,
where,
basically,
we
built
this
cube,
where
we
could
get
the
queries
down
to
six
seconds.
This
is
about
10
years
ago
now
so
in
which
the
source
of
the
data
was
actually
multi-petabytes
in
a
in
thousands
of
nodes,
hadoop
cluster,
and
then
we
end
up
processing
the
largest
cube
on
the
planet,
24
terabytes.
B
Why
do
we
do
that
for
precisely
the
same
reasons?
Actually
that
we're
talking
right
now
right
there
was
this
idea
that,
for
a
segment
of
the
queries
yeah,
we
were
just
going
to
use
the
analysis
services
cube
to
go
ahead
and
deliver
that
result
because
it
could
deliver
it
in
seconds
as
opposed
to
hours
or
never
delivering
the
answer
at
all
right
versus
okay.
Now
I
really
need
to
go
to
the
details
yeah.
B
This
will
take
hours,
but
at
least
you
can
get
it
done
as
opposed
to
never
getting
it
done
right,
which
was
then
using
hadoop
right.
That's
why
we
would
do
that
now.
Obviously,
our
technology
has
changed
over
time,
but
the
reason
I'm
bringing
up
the
scenario
is
is
not
to
recommend
using
analysis,
services
and
and
hadoop.
No,
that's,
definitely
not
the
reason.
I'm
saying
this.
The
reason
I'm
saying
this
is
because
the
context
is
that,
even
though
the
technology
has
changed,
our
reasoning
has
it
right.
B
You
don't
want
it
to
be
all
of
the
data
right
and
there's
a
scenario
where
you
absolutely
need
to
go
ahead
and
look
at
all
the
data
right
and
those
those
two
rarely
intersect,
because
it
rarely
makes
sense
to
say
give
me
in
in
like
less
than
a
second
all
of
the
data
right,
just
because,
okay,
it's
great
you
got
an
answer
except
it's
not
very
useful
to
you
right.
So
that's
more
of
the
point
right
so
relevant
of
how
the
technologies
have
changed
and
how
we
sped
things
up
and
all
that
stuff.
B
There
are
definitely
going
to
be
times
where
a
caching
layer
is
appropriate
and
there
are
definitely
times
where
the
caching
layer,
especially
if
it's
fast
changing
data,
it's
not
going
to
be
appropriate
and
that's
okay,
so
just
understand
how
to
categorize
which
data
is
which
so
that
way,
you
can
go
ahead
and
organize
it.
Accordingly.
C
And
a
lot
of
the
time,
it's
comes
back
to
that
we're
saying
last
time
the
whole
real
time
like
do
you
need
to
be
real
time.
It's
a
lot
of
it's
the
same
kind
of
because
you've
got
that
cost
on
the
cash
and
as
you're
saying
sometimes
the
cash.
It
isn't
that
useful
and
it's
like
a
do.
You
need
to
cash
all
of
it.
Do
what
you
need.
You
need
to
get
that
much
in
and
it's
yeah
it's
it's!
It's
roi
right.
B
C
B
B
Yeah
it,
it
is
exactly
the
point
like
off
to
people.
What
do
they
think
about
that
is
like
well.
Do
you
have
unlimited
money
and
unlimited
resources?
Sure
maybe
it's
possible
then,
but
the
things
that
last
time
I
checked,
you
don't
have
unlimited
money
in
the
limited
resources
number
one
and
number
two
again
it
it
allows
you
to
by
not
thinking
about
it,
allows
you
you're
being
being
a
little
harsher,
but
it's
almost
like
being
lazy
right.
B
B
I
know
you're
an
azure
guy
more
than
an
aws
guy,
so
I
can
probably
possibly
go
with
this
question
that
I
just
saw
on
youtube
live
one
of
the
top
five
biggest
differentiator
differentiators
between
azure
bricks
versus
databricks,
on
ids,
and
I
can
definitely
start
if
you'd
like,
and
then
you
can
color
code,
it
sure.
Okay.
So,
honestly,
there
isn't
much
difference
between
the
two,
so
it's
actually
very,
very,
very
ecosystem,
driven
right,
so
like
what's
a
big
difference,
how
you
set
up
your
vpc,
slash,
vnets
security.
B
A
B
Which
which
cloud
environment
which
cloud
ecosystem
you
really
want
to
use
and
for
those
environments
yeah
like
they're,
going
to
be
things
like
okay?
Well
now
I
have
to
care
about.
I
am
security
rules.
Well,
that's
obviously
always
specific
or
aed
security
principles
or
how
you
set
up
your
v-net
or
express
routes
or
whatever
else
right,
but
you'll
notice
that
my
answers
are
very
infrastructure,
related,
not
environment,
related
right,
you
can
use
for
sake,
argument
sagemaker
in
aws,
or
you
can
use
azure
ml
on
azure,
okay.
Well,
but
those
are
your
differences
right.
B
The
differences
are
more
the
which
cloud
environment.
Are
you
more
comfortable
with
and
in
reality,
it's
getting
more
and
closer
and
closer,
where
it's
just
not
which
one
you're,
which
one
you're
comfortable
with
you're
gonna,
have
to
be
comfortable
with
more
than
one.
So
it's
more
about
like
okay.
Well,
then,
how
do
you
simplify
using
the
various
components
on
those
cloud
environments?
More
than
anything
else,
so
at
least
that's
that's
my
answer,
real
quick.
I
mean,
I
don't
know
if
there's
anything
you'd
like
to
add
to
that
simon.
C
I
mean
honestly,
I
don't
have
too
much
clarity
right,
just
because
I
very
rarely
work
in
the
adb
s,
ecosystem,
there's
one
or
two
things
that
occasionally
from
an
infrastructure
point
of
view,
you'll
see
pop-up
in
one
and
then
balance
out
to
the
other
a
few
months
later
right.
I
think
the
only
example
I've
got
of
that
is
spot
vms
right.
I
think
you
can
do
that.
Currently
in
aws
it's
coming
in
azure
well
later
this
year,
then
that's
the
ignite.
C
You
know
so
that
kind
of
ability
to
say
I've
got
a
job
honestly,
I
don't
mind
if
it
just
disappears
halfway
through
and
it
comes
back
because
it's
not
priority,
but
it's
nice
and
cheap.
That
kind
of
thing,
but
that's
an
infrastructure
related
right.
That's
because
that
kind
of
functionality
was
available
in
aws.
It
got
implemented
it's
coming
in
azure,
so
it'll
get
implemented,
and
it's
like
how
it
can
take
advantage
of
that
underlying
layer,
but
the
engine
itself.
B
C
B
B
C
What
happened
because
I
was
using
autoloader,
okay,
so
autoloader
all
sorts
of
cool,
essentially
in
in
the
azure
side,
it
uses
a
thing
called
event
grid
right.
So
it's
basically
a
a
an
event
watcher.
You
know,
so
you
have
your
blob
file,
you
land
a
new
file
in
there.
It
goes
got
a
new
file
and
it
kicks
off
a
load,
and
it
starts
running
some
stuff
specifically
using
event.
C
C
B
Yeah,
so
exactly
when
you
what
what's
interesting
and
there's
a
question
that
came
up
on
linkedin,
but
I'm
not
sure
how
to
how
to
answer,
because
I'm
not
it's
a
little
too
vague,
but
but
it
actually
goes
exactly
with
what
you're
talking
about
simon
when
you're
dealing
with
situations
that
are
much
closer
to
real
time
or
at
least
near
real
time,
right
streaming
style
where
and
the
volume
is
so
high
right.
Exactly
to
your
point,
then,
all
sudden
things
like
how
the
underlying
disk
works
starts
impacting
you.
B
This
gets
us
into,
for
example,
the
small
file
problem
right,
the
the
massive
amount
of,
if
you
have
lots
of
little
smile,
a
little
smile,
little
files
right.
What
ends
up
happening
right
that
that,
basically
it
it
allows
you
to
optimize
the
rights,
especially
for
your
streaming
application,
but
it
certainly
doesn't
allow
you
to
optimize
your
reads.
So
then
you
actually
have
to
do
file
compaction
of
some
type
in
order
to
be
able
to
read
those
right.
B
So
so
that's
why,
when
you
run
streaming,
applications
you're
typically
floating
this
stuff
into
memory,
so
that
way,
you're!
Actually,
you
don't
actually
have
to
hit
the
iops,
the
disk
io
excuse
me
or
storage
io
that
comes
into
play
every
time,
you're
trying
to
read
the
stuff
right
off
the
disk
right.
B
It's
just
there
right
so
fun
stuff
like
that,
the
the
those
all
come
into
play
massively,
and
so
in
the
end,
at
the
the
deeper
you
go
into
running
streaming
applications,
and
even
though,
even
though
it's
just
auto
loader,
it's
actually
starting
to
go
down
that
route
right.
The
more
likely
you're
going
to
have
to
be
cognizant
or
aware
of
the
underlying
storage
system
that
you're
playing
with
so.
C
But
it's
I
think
there
was
a
question
about
there's
two
questions
actually
to
hook
into
that,
so
it's
all
about
optimizing
and
vacuum
and
how
often
you
run
it,
we
can
come
on
to
that
in
a
sec
and
that's
the
last
of
one
of
the
benches.
But
what
about
leaving
a
cluster
on
when
you're
streaming
is
that
kind
of
a
requirement?
I
think
I
was
in
there
somewhere?
Okay
and
that's
that's
the
interesting
pattern
I've
been
doing
recently
is
the
trigger
one
streaming
process.
Oh.
C
Yeah,
it's
doing
the
essentially
most
of
the
most
people
write
batch
queries
right.
Most
people
need
to
process
data
hourly
daily,
whatever
it
is,
and
we
kind
of
get
you
go
down
a
rabbit
hole
of
trying
to
work
out.
How
do
you
programmatically
say
just
get
the
files
I
need
just
just
get.
I
have
a
process
up
to
there.
Maybe
you
watermark
it.
C
Does
a
micro
batch
and
then
goes
unfinished,
and
so
you
know
we're
using
that
at
the
moment
to
say
get
anything
that's
come
in
since
I
last
ran,
and
so
previously
we
had
like
all
the
frameworks
saying.
This
is
how
you
work
out
the
files.
Okay.
This
is
the
file
pattern
to
match
for
files
for
this
hour,
pass
a
parameter
and
look
for
this
file
and
all
that
kind
of
stuff.
Now
it's
just
run
it
and
it'll
get
anything.
It
hasn't
got
yet,
and
so
you
can
run
it
hourly.
You
can
want
a
daily.
C
You
can
run
it
once
a
month
and
it
will
still
just
go
I'll
just
get
anything
that
came
in
since
I
last
did,
but
you're
still
dealing
with
all
those
streaming
problems,
so
we're
basically
doing
we've
essentially
taken
a
batch
process
and
added
streaming
problems
just
to
you
know,
make
life
interesting,
but
if
you're
worried
about
having
us
using
streaming
but
not
having
the
the
cluster
turned
on
all
the
time,
you
can
do
that
kind
of
pattern
just
to
go.
I
still
want
to
stream.
C
I
still
don't
have
things
coming
in
and
landing
in
an
event
hub
turn
on
every
hour
and
just
get
everything.
I've
got
and
then
get
that
pattern
working
and
then,
if
you
then
switch
it
and
go
actually
we're
now
getting
enough
value
to
leave,
it
turned
on
all
the
time.
There's
that's
no
code
change!
That's
the
just
change
the
parameter
you're
putting
into
the
trigger,
and
the
same
thing
would
just
run
as
a
full-time
streaming.
Job
like.
B
Exactly
yeah-
and
I
I
did
want
to
add,
like,
for
example,
from
summit-
I
want
to
say,
like
two
years
ago
now,
comcast
actually
went
on
stage
to
talk
about
their
environment
and
it's
exactly
this
trigger
once
that
you're
talking
about
right,
they
basically
because
they
had
converted
their
batch
jobs
into
streaming
jobs.
What
they
literally
did
was
go
from.
B
I
almost
say
if
I
recall
the
numbers
correctly
84
batch
jobs
that
they
had
to
maintain
down
to
just
three
streaming
jobs
in
essence
and
then
because
they
were
able
to
get
down
to
that,
it
was
easier
to
maintain,
but
even
better
still,
they
went
from
something
like
640
vms
down
to
64.,
so
they
saved
a
massive
amount
of
resources
and
time
because
they
so
even
though,
ultimately
they
had
an
always-on
scenario.
They
actually
were
saving
massive
amount
of
money
in
the
process
for
doing
that
right.
B
So
there
there
there's
actually
it's
actually
powerful
and
then
actually.
This
is
a
very
apropos.
We'll
probably
answer
this
one
other
question
before
we
go
into
date
and
ai
stuff.
But
how
do
you
manage
backfills?
In
that
case
right?
It
just
came
through
on
on
youtube
and
that's
actually
extremely
appropriate
right,
and
so
this
is
by
the
way
why
you'll
often
hear
us
talk
about
the
medallion
architecture.
You
know
broad
and
silver
gold.
B
The
idea
is
that,
because
you
built
your
pipelines
to
in
which,
because
you're
doing
trigger
once
and
you're,
using
that
cue,
you
event
grid
or
whatever
it
is
that
you
want
to
do,
for
you
know
for
your
for
your
in
essence,
your
queue
mechanism,
you
don't
actually
have
to
move
the
files,
okay,
so
traditional
systems
more
like
okay,
I
take
the
files
I
copy
them
over
to
indicate
that
I
need
to
go
reprocess.
Instead,
your
backfills
is
more
like
okay.
B
If
the
streaming
job
shuts
down,
okay
turn
it
back
up,
it
has
a
checkpoint,
it
goes
back
and
grabs
from
the
last
time.
You
actually
need
the
files,
but
then
the
reason
I
brought
the
medallion
architecture
is
like
how
about,
if
there's
actually
errors
in
the
silver
and
gold
like
as
you're,
trying
to
filter
augment
as
you're
trying
to
aggregate
or
build
your
features,
because
you've
got
all
the
original
data.
B
B
Obviously,
you're
gonna
need
to
knock
up
the
number
of
nodes
in
order
to
catch
up
faster
right,
but
that's
a
temporary
process
knock
it
up
process
once
you're,
good
roll,
it
back
down
and
you're
back
up
and
running
again
right
and
that
process
now
is
more
complicated
to
do
in
batch,
then
versus
streaming,
because
from
the
standpoint
of
streaming
all
you're
doing
is
like
okay,
just
you
aim
from
a
new
starting
point:
that's
all
you're
doing
right!
So
it's
it's
an
important
call.
We've
made
it
more
complicated.
C
A
C
If
it's
a
duplicate,
I
don't
care
I've
seen
it
before.
I
don't
care
if
it's
an
update,
you're
just
chunking
this
data
exactly
yeah
and
that's
just
and
we're
streaming
delta
streaming.
So
raw
is
a
delta
table.
So
we
insert
appending
into
there
we're
doing
stream
from
a
delta
table
into
a
middle
tier
silver
base.
Whatever
you
want
to
call
it
delta
table
and
did
a
video
on
it
yesterday,
you
can
do
merging
on
a
stream.
C
You
know
so
you
can
say
micro
patch,
for
each
batch
do
a
delta
merge.
So
actually
you
can
just
drop
a
new
file
and
even
if
you've
loaded,
that
file
previously
it'll
work
out
the
merging
and
it'll
just
kind
of
slot
those
files
in
now
not
the
most
performant
for
doing
a
high
volume
lined
live
stream
right,
but
for
doing
the
trigger
once,
if
you're
just
saying
just
use
this
mechanism
and
actually
just
turn
on
and
just
churn
through
anything,
that's
come
in.
C
B
B
B
B
C
C
Which
is
essentially
some
of
this
stuff,
how
you
model
things
for
the
end
user?
So
not
the
engineering,
not
the
getting
data
in,
but
actually
what
are
the
things
that
you
can
do
because
of
spark
3
because
of
adaptive.
Query:
execution
dynamic
partition
processing
because
of
some
of
the
delta
functions.
How
do
you
actually
get
the
sql
people
a
good
plane
to
actually
work
on.
B
B
Bam
that
was
beautiful,
definitely
take
a
look
at
his
session.
It's
awesome,
especially
for
our
european
friends,
because
it
is
designed
for
that
time,
but
it
will
be
available
on
demand
if
you
don't
want
to
wake
up
three
in
the
morning.
Admittedly
enough,
that's
my
not
my
favorite
thing
to
do,
but
there
you
go
okay
and
then,
in
in
my
case,
I'll
freely
admit
that
I
forgot
the
name
of
our
title:
two.
Okay,
this
is
myself
and
barack
obvious.
B
We
we
had.
We
either
called
it
under
under
unpacking
the
transaction
log
v2
or
under
the
sediments
v2.
I
forgot
which
one
we
chose,
but
it
is
a
technical
dive
into
how
the
transaction
log
of
delta
lake
works.
This
is
not
obviously
we're
showing
demos
on
databricks,
but
this
is
not
databrick
specific.
This
is
very
much
into
how
the
open
source
delta
lake
works.
We
dive
into
what
the
v2
checkpoint
works.
B
This
is
oh,
it's
a
follow-up
to
our
unpacking,
the
transaction
log
tech
talk
that
we
did
originally,
but
it's
it
follows
up
with
the
v2
design.
So
it's
it's
not
just
v2
of
the
video
or
the
tech
talk.
It's
actually
v2
of
the
actual
checkpoint
design
itself
as
well.
So
it's
actually
a
fun
play
on
words,
so
so
there's
that
session
we
also
have
three
fun
ama
sessions,
as
karen
had
called
out
in
the
beginning,
I'll
be
emceeing
two
of
them.
B
One
will
be
for
the
for
vips,
specifically
on
lake
houses.
Okay,
the
both
the.
If
you
want
to
think
about
from
the
concept
or
the
paradigm
of
it,
the
other
part
absolutely
is
if
you
want
to
dive
deeper
into
technical,
the
other
one
that
I'm
seeing
is
absolutely
on
the
photon
engine
itself
and
the
delta
lake
itself
and
understanding
the
technical
details.
B
B
If
you
have
not
already
read
his
book
outliers,
do
it
another
good
book
for
if
you
are
a
geek,
and
you
don't
like
talking
to
people
talking
to
strangers,
that's
a
great
book
as
well,
and
the
reason
I'm
promoting
him
is
because
he's
a
fellow
canuck.
That's
right
go
canadians,
but
nevertheless,
also
if
you
are
don't
want
to
read,
but
rather
listen,
he's
got
an
amazing
podcast
series
called
the
revisionist
history.
It's
an
excellent
podcast
series.
I
highly
highly
recommend
that
it's
probably
a
lot
more
organized
than
myself
and
simon.
B
A
Thanks
cindy
thanks
for
adding
on
some
added
details
on
summit
that
denny
and
and
is
way
more
involved
in
summit
with
sessions
and
whatnot,
so
I'm
glad
he's
able
to
share
a
little
bit
more
about
what
we
have
going
on
for
you
all.
We
also
have
three
meetups
two
they're
on
the
data
plus
ai
online
meetup
group
page.
So
if
you're
interested
in
joining
us
there
as
well,
I
also
see
I
have.
I
just
checked
my
email.
A
I
have
a
few
folks
that
emailed
me
for
vip
passes,
which
I'm
excited
about,
so
I
do
have
a
few
left,
so
email
me
or
message
me
in
linkedin
and
I'll.
Send
you
one
your
way,
we'd
love
to
have
you
there
with
that.
Thank
you
so
much
for
your
time,
simon
and
denny,
we'll
let
you
get
back
to
your
session
planning
for
summit
and
thanks.
Everyone
for
joining
us
take
care
have
a
good
day.