►
From YouTube: Simon Whiteley + Denny Lee Live Ask Me Anything
Description
Simon and Denny Build A Thing is a live webshow, where Simon Whiteley (Advancing Analytics) and Denny Lee (Databricks) are building out a TV Ratings Analytics tool, working through the various challenges of building out a Data Lakehouse using Databricks. In this session, they'll be talking through their Lakehouse Platform, revisiting various pieces of functionality, and answering your questions, Live!
This is your chance to ask questions around structuring a lake for enterprise data analytics, the various ways we can use Delta Live Tables to simplify ETL or how to get started serving out data using Databricks SQL. We have a whole load of things to talk through, but we want to hear YOUR questions, which we can field from industry experience, community engagement and internal Databricks direction. There's also a chance we'll get distracted and talk about the Expanse for far too long.
Connect with us:
Website: https://databricks.com
Facebook: https://www.facebook.com/databricksinc
Twitter: https://twitter.com/databricks
LinkedIn: https://www.linkedin.com/company/data...
Instagram: https://www.instagram.com/databricksinc/
B
A
Everybody
this
is
simon
and
simon
right.
There
and
denny
live
ama
because
we
are
alive
and
we
we're
actually
here
to
go
ahead
and
talk
about
it's
a
basic
extension
of
our
build,
a
thing,
webinar
series
you
may
or
may
not
recognize
realize
this,
but
this
is
in
fact
simon
and
my
first
time
actually
working
together.
Physically
we've
actually
only
worked
together
online,
virtually
this
entire
time.
So
we
want
to
make
this
an
interactive
session.
So
we
would
like
you
all
to
actually
ask
questions.
A
It
allows
us
to
be
lazy
too,
so
it
allows
us
to
achieve
both
things,
but
in
all
seriousness
we
hope
you
actually
have
lots
of
questions
when
it
comes
to
delta
lake
delta,
live
tables,
autoloader
things
of
that
nature,
simon
and
his
team.
The
cohort
that's
sitting
up
here,
basically,
terry,
okay,
sorry,
basically
they
are
the
awesome
consultancy
that
actually
has
worked
with
databricks
and
other,
oh
sorry,
other
yeah,
okay,
oh
oh
right,
there's
a
camera
there
right,
okay,
and
so
there
and
other
other
systems.
A
When
it
comes
to
big
data
systems.
Okay,
they're
passed
as
he's
currently
trying
to
log
in
because
apparently
he
has
no
internet
access
whatsoever
is
in
fact
actually
from
the
sql
server
years.
Actually,
so
they
actually
know
a
lot
about
database
and
bi
systems.
In
fact,
that's
actually
how
come
simon
and
I
actually
got
along
well
because
I'm
actually
formerly
sql
server.
A
A
Yeah
all
right
all
right,
all
right
so
anyways,
that's
me
going
ahead
and
just
holding
the
fort
and
so
but
nevertheless
okay.
So
we
do
want
to
talk
about
build
a
thing.
So
in
essence,
if
you
have
your
questions
when
it
comes
to,
like
I
said,
delta
delta
light
tables
autoloader
anything
else,
we
should
include.
D
A
D
A
D
You're,
the
professional
okay,
so
we
did
a
series
of,
I
think
five
videos
we
went
through
going
right
are:
we
gonna
just
demonstrate
how
to
put
a
lake
house
together.
So
a
lot
of
that
is
a
what's
a
lake
house,
all
that
kind
of
stuff.
If
those
are
questions
you
guys
have,
we
can
talk
to
them,
but
we
built
a
little
tiny
scraper
that
went
and
talked
to
a
web
api
got
some
data
down
and
put
it
into
a
landing
area
that
was
kind
of
just
do
a
quick
thing.
D
D
So
we've
been
through
that
process
just
to
kind
of
show,
it's
fairly
easy.
It's
failure
to
set
this
stuff
up,
it's
value
to
go
through
the
process
and
build
out
essentially
a
nice
sql
warehouse
layer,
but
it's
in
the
lake
and
you
can
go
and
build
it
from
there.
So
we
thought
for
this
session,
we'll
just
open
the
floor
to
an
ama.
If
there's
any
of
those
bits
that
you
want
to
dig
into
and
go,
actually
I
didn't
bother
watching
the
series.
What's
that,
how
does
that
work?
We
can
talk
about
it.
D
A
E
Yes,
so
I'll
embarrass
julie
here
we
work
together
we're
working
on
a
consulting
project
for
credit
union
and
we're
working
with
form
data.
That's
xml
we're
using
azure
data
factory
to
put
it
in
cosmos
db
and
there's
a
spark
connector
for
cosmos
db
to
load
it
into
databricks
and
we're
running
to
some
performance
issues.
E
A
A
Now
it's
my
understanding
with
the
cosmos
db
team
that
they
actually
have
a
version
of
it.
That's
supposed
to
work
with
spark
3.x,
but
it's
not
public
yet,
but
there
are
specific
customers
to
my
understanding
that
are
currently
using
it
by
the
way,
I'm
formerly,
of
the
cosmos
db
team.
So
that's
the
reason.
I
know
all
this
stuff.
A
Okay,
so
now
saying
all
that
the
key
key
concern
that
you're
still
going
to
have
when
you're
extracting
data
out
of
even
if
you're
using
the
latest
version
of
the
driver,
pretend
it's
spark
3
everything's
good
to
go.
The
usual
problem
that
you're
going
to
have
is
number
one
you're
going
to
have
to
knock
up
the
number
of
reu's
to
a
really
high
number.
A
In
order
for
you
to
be
able
to
read
fast
enough
out
of
cosmos
db,
even
if
you're
trying
to
use
something
like
htap
or
whatever
else
doesn't
matter,
you
actually
have
to
have
the
ru's
at
a
pretty
high
number
in
order
to
extract
it
out.
You're
gonna
immediately
tell
me
dude,
that's
a
lot
of
money,
and
yes,
that
is
the
correct
answer.
So
what
you're
going
to
want
to
do
is
basically
knock
up
the
rus
long
enough
just
to
do
the
extraction
and
then
knock
the
rus
back
down
almost
immediately.
D
A
D
Me
I
wouldn't
try
and
query
the
whole
of
cosmos
db.
I'd
switch
on
to
the
change
feed.
You
want
to
pull
out
updates
from
cosmos
db
rather
than
try
and
hit
it
and
query
the
whole
thing
which
cost
must
be
not
greater
and
gonna
cost
you
a
lot
of
money
for
doing
a
big.
That's
true
analytical
query.
Just
take
the
the
change.
No.
A
F
A
All
right
so
either
way,
and
yes,
I
agree
with
them
fully.
You
should
hit
the
change
feed.
Okay.
Now
the
context
is
that
even
after
you
do.
That,
though,
is
that
what's
happening
is
that
the
data
may
not
be
in
the
same
partition
format
that
you
actually
want
to
put
it
into
whatever
whether
it's
as
your
data
factory
into
a
parquet
table
delta
table.
I
don't
care,
I'm
not
trying
to
pitch
one
or
the
other.
At
this
point.
A
Obviously
I
prefer
using
delta,
but
you
know
that's
just
me,
but
in
all
seriousness,
the
partitioning
scheme
that
you
actually
extract,
the
data
out
of
cosmos
and
the
partitioning
scheme
of
the
table,
or
even
just
spark
in
memory,
often
will
be
different.
So
you
may
actually
need
to
re-partition
the
data
in
memory
for
spark
to
actually
query
it
in
an
optimal
manner.
A
Okay,
so
there's
especially
with
the
change
data,
feed
that'll,
be
the
problem,
because
what
you're
doing
is
you're
actually
extracting
out
small
little
tidbits
of
transactional
data
at
any
point
in
time,
if
you're
then
trying
to,
for
example,
create
that
little
bit
with
the
historical
context,
that's
actually
completely
different.
Partitioning
schemes
typically.
E
A
Between
the
day
and
also
specifically
again
yeah
the
assignments
point,
the
change
data
feed.
I
think
that
you'll
be
able
to
reduce
the
amount
of
res.
You
actually
need
to
extract
out
the
data
and
then
what
element
happening
is
that
you'll
still
most
likely
because
you're
doing
historical
query,
contexts
right,
you'll
probably
need
to
re-partition,
but
you
can
do
that
in
memory
with
whatever
the
data
the
data
frame
is
so
that
way,
it'll
actually
speed
up
your
query
performance.
That
way.
E
D
A
C
D
I
can
take
that
from
a
design
pattern,
absolutely
yeah,
so
we
kind
of
started
implementing
a
separate
landing
area
for
a
few
reasons.
Firstly,
so
this
is
very
a
very
azure
answer
to
the
question.
If
we
can
use
it
in
blob
storage,
we
can
kind
of
change
the
security
configuration
of
it.
So
we
have
like
the
full
lake,
the
adls
gen
2
lake,
where
we
can
have
like
a
certain
level
of
replication
we
can
decide.
D
D
So
we
normally
just
land
the
original
file
type
into
landing,
and
we
get
lots
of
questions
from
people
who
are,
you
know
extracting
from
a
sql
database
or
extracting
from
a
from
a
structured
type,
and
we
go
great
okay,
you're
gonna.
Take
that
and
then
you're
gonna
remove
all
structure
and
put
it
into
a
landing
area
and
then
we'll
pick
it
back
up
and
put
structure
in
and
they're
like.
What
are
you?
Are
you
insane?
Why
are
you
doing
that?
D
And
for
me,
it's
all
about
scheme
revolution
schema
drift,
all
that
kind
of
good
stuff.
So
these
days
inside
of
autoloader,
you
can
do
things
like
the
rescue
style,
schema
evolution
so,
and
you
can
only
do
that
when
it's
reading
on
unstructured
files.
So
it
can
only
do
it
on
json
and
I
think
delimited
text
now
correct
right
now,
so
you
have
to
take
away
the
structure
so
that
you
can
compare
it
to
another
structure
to
say:
has
it
changed?
D
So
if
a
vendor's
sending
you
data
and
then
one
day
you
get
an
extra
column
or
it's
a
bigger
data
type
and
you're
protecting
yourself
against
schema
drift.
We
can
handle
that
in
a
structured
way
as
part
of
the
thing
and
that's
why
we
have
that
separation
of
that's
not
really
part
of
the
lake,
because
that
could
be
those
files
could
be
incorrupt.
D
They
could
just
have
sent
us
garbage
that
could
be
a
massive
changed
file
and
then,
when
it's
in
the
lake,
it's
because
we've
picked
it
up
and
put
it
through
a
data
frame
in
autoloader
we've
got
structure
so
that
can
be
delta
all
the
way
through
and
we've
got
all
the
good
governance.
We've
got
all
the
good
acid
transactions
and
all
of
that
stuff.
A
Yeah-
and
one
thing
I
would
actually
add
just
to
provide
some
additional
context
here-
is
that
when
it
comes
to
the
those
dmz
zones,
often
you're
not
the
one
controlling
it
right,
it's
often
controlled
by
some
other,
like
not,
every
company
operates
the
size,
obviously,
but
the
planning
companies.
What
they
do
is
that
the
dmz,
basically
the
pre
bronze
in
essence,
that's
controlled
by
it
or
controlled
by
some
other
organization
anyways.
So
the
fact
is
you
don't
actually
you
just
in
essence,
you
don't
want
to
trust
the
source.
A
You
just
assume
that
they're,
just
it's
a
dumping
ground
and
then
you
treat
it
as
a
unsafe
source
to
begin
with.
Maybe
that's
a
little
too
strong,
but
the
context
is
that
basically
you're
not
you're,
not
pretending.
The
scheme
is
correct.
You're,
not
pretending.
The
date
is
even
correct.
You're
going
to
validate
it
before
it
comes
in
so
essentially.
D
D
D
So
you
can
say
I
want
to
make
a
data
frame
that
data
frame
is
going
of
over
format,
cloud
files
and
if
it's
cloud
files-
and
that
means
it's
auto
loader
and
so
all
we've
got
inside
our
delta
live
tables
job
the
first
table.
It's
actually
software.
It's
a
view.
That's
doing
the
the
auto
load
debating
kind
of
the
demo
we
put
together.
I
think
yeah.
I.
D
A
So
what
we're
trying
to
do
is
we're
trying
to
avoid
that
particular
problem
by
basically
building
up,
in
this
case
we're
using
roxdb
to
basically
keep
the
state
of.
What's
the
requeue
and
the
delete
queue,
delete,
cube
basically
says:
okay,
I've
done
processing
that
file
now
go
delete
it,
okay
and
or
in
the
req.
I
need
to
go
ahead
and
read
that
file
so
as
opposed
to
you
needing
keep
the
t,
keep
the
state
of
all
of
these
different
files.
A
Autoloader
basically
takes
care
of
that,
for
you
all,
you
did
was
basically
specified
cloud
files,
but
basically
underneath
the
cover.
So
it's
automatically
taken
care
of
all
that.
So
hopefully
that
clarifies
that
the
reason
why
we
want
to
do
that
and
how
that's
actually
separate
from
dlt
from
an
orchestration
perspective.
B
H
B
I
I
feel
a
lot
smarter
for
watching
it
and
I'm
actually
a
databricks
employee
as
well.
So
thank
you
for
that.
So
now
I've
got
a
couple
of
questions
about
this,
so
I
build
a
lot
of
demos
and
I've
built
a
thing
exactly
like
this.
Actually
I
just
wonder
how
so
you
use
adf
to
get
your
json
data
into
into
the
cloud
data
layer.
B
Do
you
have
other
tools
that
you've
used
that
are
really
good
for
polling
rest
apis
like
random
rest
apis,
I
find
myself
doing
it
all
the
time
like
adf
is
kind
of
a
bit
of
a
heavyweight
tool.
For
that,
can
you
recommend
any
other
tools
and
then
the
second
part
of
the
question
is:
can
you
just
explain
a
little
bit
about
how
frequently
the
json
files
are
dropping
in
here
in
this
example,
and
you
know
how
many,
how
often
and
how
autoloader
handles
all
that
cool.
D
All
right
so
question
one:
no,
no!
I
mean
a
bit
of
context
there.
It's
because
most
of
our
clients
tend
to
be
azure
clients
and
most
of
our
clients
are
already
doing
some
kind
of
cross-network
shifting
of
data
they're
getting
data
from
their
corporate
network
they're
pulling
it
into
the
cloud.
They
don't
want
to
do
all
the
networking
plumbing
of
making
it
so
datavix
itself
can
see
all
their
internal
systems
to
do
a
pull
directly.
So
we
nearly
always
end
up
having
data
factory
in
our
azure
implementations.
D
Just
to
do
that,
and
if
we've
already
got
it,
we've
already
bothered
deploying
the
infrastructure.
We've
already
got
the
devops
2x
to
get
it
there.
We
just
use
it
out
of
almost
laziness
of
not
setting
up
a
different
thing
that
we
have
to
secure
differently
and
look
after
now,
there's
been
a
very
big
focus
on
lots
of
different
integration
tools
and
kind
of
five
times
being
talked
about
a
lot
exactly.
I
don't
have
any
experience.
A
With
it
yeah
so
in
general,
you
can
actually
look
at
a
lot
of
exactly
like
what
simon
called
out
like
five
trend.
Dbt
there's
all
these
other
integrations
that
actually
work
really
well,
and
for
that
matter,
honestly,
you
can
even
just
like,
especially
if
you're
just
reading
a
rest
api
to
grab
it
literally
you
just
do
a
python
calls
right
and
distribute
with
spark
for
that
matter.
Yeah
literally.
B
A
That's
true
it's
very
true,
and
but
in
the
end
it's
it's
more.
It
literally
is
more
matter
of
like
it's,
your
preference
more
than
anything
else,
just
like
what
simon's
talking
about
his
his
customers
and
his
environment
is
typically
going
to
be
doing
adf
as
it
is
in
the
first
place.
Honestly,
most
of
my
customers
they're
just
going
to
code,
it
themselves
right,
so
they're
literally
going
to
either
run
a
rust
api
or
they're
going
to
write
a
python
api
to
go
ahead
and
do
the
queries
anyways.
D
In
terms
of
how
often
you
run
this
thing,
I
mean
that's
the
nice
thing
about
so
whether
we're
using
delt
life
tables
or
whether
we're
doing
kind
of
so
we
build
a
lot
of
the
stuff
for
ourselves
and
the
big
pie
spark
script
and
we
just
run
it
through,
but
we
use
the
same
pattern.
So
we
use
autoloader
going
from
a
landing
area
to
pull
data
into
what
we
call
raw
bronze,
medallion
architecture-
stuff,
yes,
and
then
move
it
from
there.
D
We
stream
from
that
raw
table
into
like
a
cleaning
table
and
so
essentially
kind
of
that
that
area,
auto
load
of
itself
is
a
streaming
technology.
Delta
streaming
is
obviously
streaming
technology,
so
you
can
leave
that
turned
on
and
then
literally
whenever
someone
drops
a
file
in
that'll
automatically
just
get
ingested,
and
so
your
latency
you're
talking
about
a
couple
of
seconds
or
you
can
switch
it
and
say
I
want
to
use
trigger
once
or
trigger
available.
D
Now
and
that's
you
kick
it
off
like
a
batch
job
and
it
still
does
streaming
but
initializes
the
stream
works
out.
Has
any
files
come
in
since
I
last
ran,
if
so
processes
them,
and
then
it
shuts
down
the
cluster,
so
you're
not
paying
for
it
to
be
there
all
day.
But
the
nice
thing
about
that
pattern
is
so
you
maybe
you've
got
like
a
an
overnight
etl
load
in
a
good
classic,
bi
warehouse
overnight
load
and
you
you
do
that
and
you
just
kick
off
this
job.
D
Anything
that's
coming
during
the
day
you'll
process,
and
then
they
go.
Oh
actually
can
we
start
doing
an
hourly
and
you
just
change
the
schedule.
There's
no
code
change,
because
actually
it's
just
all
whatever's
happened
since
I
last
ran
and
it
just
gives
you
the
flexibility
of
you
can
run
the
same
thing.
Every
five
minutes
every
hour
every
day
with
no
code
changes
which
is
great.
A
Yeah
so,
for
example,
one
example
that
I
like,
using
especially
when
it
comes
to
structure
streaming,
especially
talking
about
that
trigger
once
concept
right.
So
for
one
examples
we
used,
I
think,
a
couple
years
ago,
pretty
heavily
was
comcast
right
and
what
happened
is
that
they
were
trying
to
take
the
set-top
box
information,
and
so
they
used
to
have
by
basically
using
structure
streaming
with
trigger
once
and
with
delta
lake.
So
they
can
ensure
transactional
reliability
of
the
data.
They
went
from
640
vms
down
to
64.
A
and
by
using
the
trigger
once
they
went
from
84
batch
jobs
down
to
three
so
and
also
their
latency
dropped
at
least
half.
So,
basically,
that's
all
they
did
they
didn't.
They
actually
changed.
None
of
their
business
logic.
They
just
simply
said
we'll,
simply
switch
the
trigger
once
and
that
basically
simplified
the
process
and
they
were
able
to
use
10x
less
vms
so
seems
like
a
pretty
good
win.
So.
A
D
Haven't
changed
it
on
my
invitation,
honestly
yeah,
I
know,
but
still,
but
it
used
to
be
if
you
use
trigger
once
that,
would
try
and
load
everything.
That's
happened
since
as
a
single
micro
batch,
which
isn't
the
most
efficient
thing
in
streaming,
just
something
that
there's
a
massive
peak
of
load
of
data
was
available.
Now
actually
goes
right,
I'm
going
to
start
I'm
going
to
keep
doing
micro
batches
until
I've
caught
up
and
then
I'm
going
to
stop.
D
G
If
you
want
something
lighter
weight,
then
data
factory
and
azure
functions
is
also
a
good
choice.
Yeah
azure
functions
you
just
write
about
the
code
to
do
it
yeah
we
use
it
where
you
can't,
where
you
fall
off
the
edge
of
what
data
factory
can
do
and
then
you
have
to
do
something
custom
in
a
function,
so
you
can
just
do
it
with
a
function.
Yeah.
A
That
one,
I
actually
do
not
know
the
answer
to
mainly
because
I
know
we
were
supposed
to
be
updating
autoloader
to
actually
provide
additional
monitoring
reporting.
But
I
don't
know
the
current
state
now
in
terms
of
the
actual
information
in
terms
of
it's
already
in
the
roxdb
database
itself.
So
in
other
words,
you
can
actually
track.
Oh
exactly
what
the
requeus
and
what
the
delete
queues
are
exactly
at
that
time.
I
just
don't
know
what
the
current
state
is
in
terms
of
like
how
much
the
reporting
is
coming
out.
So.
G
A
D
Answer
and
from
our
less
technical
just
build
it
into
the
process
thing
we
just
automatically
sample
records
going
through
into
the
raw
area
with
the
input
file
name.
So
actually
we
can
just
do
a
select,
distinct
of
all
the
different
files
so
that
we
get
our
own
list
not
as
good
as
being
able
to
pull
it
straight
from
metrics
logs
yeah,
but
as
part
of
the
process.
We
know
what's
been
loaded
and
we
can
always
tally
up
and
go
that's
what
I
think
I've
loaded.
D
A
D
But
it's
honestly
a
conversation
I
have
with
clients
when
we
say
hey.
Look.
We
love
this
autoloader
approach
because
you
no
longer
have
to
wonder
about
you
know,
running
a
job
and
saying
look
for
files
that
match
this
kind
of
file
pattern
and
building
a
whole
watermarking
metadata
thing
to
try
and
work
out
what
files
you
need
to
load.
You
get
rid
of
all
of
that
complexity,
but
you
do
add
other
complexity.
Other
things
are
slightly
harder,
reloading
a
file
which
comes
up
occasionally
yeah.
D
It's
not
that
elegant
to
pattern,
but
that's
kind
of
one
of
the
things
that
you're
getting
by
saving
all
the
other
time
in
terms
of
doing
that,
file
matching
etl
stuff.
No,
that's
fine!
So
there
are
one
or
two
things
that
become
more
complicated
if
you're,
letting
it
do
black
box.
If
you're
saying,
I
trust
you
to
manage
things
for
me,
occasionally
you
need
to
kind
of
just
play
the
game
and
reload
the
file.
Yes,.
A
That
much
more
clear
is
that
what
you're
asking
me
so
you're
just
doing
it
live
yes,
okay,
cool
all
right,
fair
enough!
All
right,
that'd
be
fun!
No,
no!
That's
fine!
I'm
cool
with
it!
I'm
cool
with
it!
It's
okay,
since
the
product
management
team.
Isn't
here
anyways
I'll,
just
forget
it
anyway!
So
it's
okay!
Well,
you
are
databricks.
Am
I
no?
No
look
at
me.
Look
at
the
shredder.
H
A
Yeah
I
mean
it
simplifies
it.
Basically,
that's
the
key
thing.
So
sorry,
just
to
repeat
the
question,
so
everybody
here's
it.
Basically,
the
main
advantage
of
autoloader
is
that,
basically
you
don't
have
to
do
worry
about
the
file
listing,
so
you
can
in
essence,
for
example,
if
you
wanted
to
do
this
yourself,
basically
off
the
storage
system,
there's
a
basically
there's
a
queue
mechanism
that
can
give
notifications
of
when
a
file
drops
into
the
into
the
storage
system.
A
You
can
in
essence,
then
take
that
notification
store
that
in
a
queue
of
some
type,
to
basically
track
which
files
that
you
need
to
process.
Okay
and
then,
once
you
go
ahead
and
finish
processing
that
file,
then
you
can
go
ahead
and
remove
that
from
the
queue.
So
that
way,
you
know
that
you've
already
processed
the
file
or
you
can
let
autoloader
do
everything.
I
just
said.
D
And
I
mean
if
you're,
if
you're
treating
it
as
like
a
daily
load
thing,
that's
like
a
tiny
advantage
right,
yes,
correct!
It's
it's
going
to
take!
Oh,
it's
not
going
to
take
10
seconds,
doing
a
file
listing
it's
going
to
take
a
millisecond.
That's
not
that
big,
an
advantage
sort
of
loader
itself
on
that
purity.
You
know
it's
great
if
you're
streaming,
it
means
you're,
micro,
batches
and
you're,
constantly
doing
it
way
faster,
but
for
the
kind
of
daily
load
trigger
once
thing.
D
It's
not
adding
that
much
from
just
the
file
notifications,
but
actually
it's
then
kind
of
built
on
top
of
autoloader.
You've
got
a
load
of
schema
management
so
actually,
when
it
runs
it'll
infer
the
schema
and
it
can
land
a
file
that
actually
has
a
schema
metadata,
so
you're
not
having
to
have
a
separate
metadata
thing
and
work
it
out
and
keep
it
yourself
and
tell
it.
What
structure
to
have
it's
just
going
to
manage
it
as
metadata
next
to
the
file?
And
then
it's
got
some
mechanisms
in
there
to
go.
D
The
data's
changed,
let's
kind
of
stop
and
restart
it
and
update
it.
All
the
data's
changed:
let's
dump
it
into
an
unstructured
column,
so
people
can
query
the
extra
data
without
having
to
change
the
actual
physical
structure
so
started
off
just
as
a
pure
optimization
for
that
file
notification
queue.
Now
it's
got
a
few
bits
of
functionality
baked
on
top
of
it
that
just
make
any
kind
of
ingesting
weird
and
wonderful
data
from
files
just
a
little
bit
better
a
little
bit
simpler.
Yes,.
A
Exactly
yeah
and
basically
the
most
common
approach.
Basically,
is
that
rescue
one
where?
Basically
because
the
schema
has
changed,
but
you
still
want
the
process
to
continue
running
you
don't
want
it
to
stop.
Then.
Basically,
there's
literally
this,
the
column
literally
by
default,
called
underscore
rescue.
So
you
basically
query
that
column.
That's
where
the
unstructured
semi-structured
data
actually
sits.
D
A
And
neither
neither
option
is
valid
right.
I
mean
there
are
definitely
times
where
it
makes
sense
to
make
this
change
schema.
There
are
definitely
times
where
you're
going
no
way.
This
is
complete
garbage
right
because
I
actually
received
the
wrong
data
sets
all
together
right.
So
the
whole
point
is
that
we
we
simplify
that
process.
So
that
way
you
allow
the
process
to
continue
running
this
again.
This
is
even
if
you're
talking
about
batch
processing
but
just
frequent
batch
processing.
A
The
whole
point
is
that
you
don't
want
the
job
itself
to
stop
in
a
production
environment.
You
want
it
to
continue
running,
and
so
this
the
giving
now
in
the
end
the
important-
and
I
think,
part
of
the
reason
why
we
keep
talking
about
this
stuff
is
because
it
really
does
depend
on
what
your
business
logic
is.
A
D
D
So
people
can
go
in
and
query
it,
so
it's
great
from
analysts
going
actually
is
there
any
value
in
this
new
random
thing
again?
Oh,
that's,
really
useful
I'll
bring
it
into
a
report
right
and
then
they
go.
Can
we
get
this
permanently
and
then
you
need
to
make
a
physical
update
to
the
delta
table
and
say
this
is
now
a
first
party
part
of
my
table
yeah,
but
it
means
you've
got
control
about
when
you
do
that,
it's
now
a
process
about.
A
F
F
So
I
have
a
very
specific
question.
Maybe
this
is
too
detailed,
so
we
can
do
this
offline
as
well,
but
when
it
comes
to
spark
structured
streaming,
what
my
observer
observation
has
been
that
when
does
the
spark
behavior
change,
depending
on
the
sync
that
you
have
so
in
my
example,
what's
what's
been
happening?
Is
I
have
a
delta
table?
I
use
that
as
a
stream
and
then
load
it
if
I'm
loading
it
into
another
delta
table
using
max
files
for
trigger
equal
to
one
setting.
F
A
Okay,
so
there's
no
simple
answer
to
this
question.
Okay,
so
let
me
so
let
me
try
to
at
least
like
bubble
it
up
to
a
more
generic
question.
Okay,
when
it
comes
to
using
any
streaming
system.
This
is
not
specific
to
structure
streaming.
It's
obviously
applicable
that,
but
I
can
I
can
give
you
the
same
answer
for
flank.
I
can
give
you
the
same
answer
for
pulsar.
This
is
not
a
knock
knock
on
any
system.
This
is
just
saying
in
general,
when
you
have
streaming
systems
of
any
type.
A
The
idea
of
writing
to
one
storage
versus
the
other
is
going
to
result
in
at
times
very,
very
different
contexts
and
performance
structures.
Okay,
so,
for
example,
structure
streaming
to
a
delta
table.
Yes,
that
is
beyond
optimized,
because
that's
frankly
how
we
built
it
okay,
so
that
is
relatively
straightforward.
The
reason
why
is
because
it's
writing
to
a
parquet
file
in
the
first
place,
which
is
what
spark
was
designed
to
do
in
the
first
place?
We
just
simply
added
transaction
semantics,
so
we
can
ensure
the
transaction
reliability
of
the
data.
Okay
bam.
A
A
Okay
problem
number,
one
exactly
when
we
talk
about
exactly
what
semantics
we're
instructor
streaming
exactly,
what
semantics
means
from
the
source
at
the
point
in
time
that
we
read
it
to
the
point
in
time
where
we're
just
about
to
write
it,
but
we
actually
haven't
written
it
yet
remember
it's
in
the
data
frame
we're
about
to
write
it!
That's
the
exactly
one
semantics
I'm
talking
about,
in
other
words,
when
we,
when
we
process
the
data
we
when
we
see
it
once
we
only
process
it
once.
If
we
see
it
again,
we
won't
process
it
again.
A
Okay,
that's
what
we
mean
by
exactly
one
semantics
now,
there's
the
problem
of
actually
writing
it
to
the
end
state.
Okay,
now
again,
because
spark
is
designed
really
well
to
write
to
parque
in
the
first
place,
having
to
write
to
delta
and
actually
to
give
credit
to
my
competitors,
okay,
to
iceberg
to
hootie,
it's
designed
for
that
purpose.
Okay.
A
Well,
if
your
data
for
sake
argument
is
partitioned
by
just
to
oversimplify
the
problem
partition
by
date,
time,
okay,
so
in
other
words,
you've
got
5000
rows
and
they're
a
partition
by
the
hour,
let's
just
say:
okay,
but
the
document
store
itself
is
partitioned
by
country
for
the
for
the
fun
of
it.
Okay,
that
alone
is
basically
going
to
force.
One
stream
to
take
on
the
vast
majority
of
the
rights
and
the
rest
of
streams
are
basically
running
idle.
Okay,
so
each
one
of
these
environments,
there's
going
to
be
two
things,
are
going
on.
A
One
is
what
is
actually
the
partition
key
primary
key
document
id
whatever
system
it
is
that
allows
for
that
distribution.
The
second
thing
is
definitely
the
configuration
itself,
so,
in
other
words,
just
like
the
original
question
about
cosmos
db,
if
you're
using
a
spark
2.4
but
you're
but
you're,
actually
a
a
connector,
but
it's
actually
running
on
spark
3.3
yeah
aqe
configurations
are
going
to
completely
screw
this
entire
thing
up.
A
So
even
if
it's
trying
to
write
to
delta,
it's
still
going
to
screw
it
up,
because
we
were
trying
to
actually
distribute
the
data
in
one
particular
way,
and
it
completely
doesn't
work
right.
Then
we're
trying
to
go
ahead
and
write
to,
like
you
know,
a
cassandra
or
whatever
else
well
again,
which
version
of
java,
which
version
scala.
Okay,
if
you're
using
scala
2.11,
which
a
lot
of
systems
are
using,
that
means
I
can
only
handle
up
to
12
columns.
A
Well,
sorry,
24
columns,
anything
above
that
also,
basically
it's
paging
in
memory
it
might
be
able
to
handle
the
36
column,
but
guess
what?
Now?
Basically,
the
driver
is
full
of
memory.
It's
ooming
to
death
and
basically
your
performance
just
drops
massively
and
I'm
going
to
stop
now,
because
I
just
realized
I'm
rat
holding
on
to
hell.
A
But
the
point
is
that
every
single
one
of
these
things
are
going
to
act,
a
version
of
what
I
just
said:
okay,
so
as
much
as
I'd
love
to
say,
there's
a
simple
answer,
as
you
can
tell
how
I've
already
rattled
beyond
belief,
yeah
it's
complicated
as
heck,
and
so
that's
the
main
reason.
Why?
Because
the
way
a
doc
store
works
in
the
way
of
file
system
works
are
inherently
different
and
how
you
optimize
for
them
are
also
inherently
different.
A
A
A
That's
right,
I'm
sorry!
Okay,
I
I've
sorry
spark3.x
introduced
a
concept
called
aqe
adaptive,
query
execution,
so
the
whole
purpose
of
adapter
query
execution
as
part
of
the
spark.3.x
line-
is
that,
as
opposed
to
when
you
create
a
logic
when
you
look
at
the
catalyst
center
for
spark
there's
what
they
call
first,
initially
the
logical
engine
and
the
physical
engine,
okay,
so
the
logical
engine
basically
builds
up
a
bunch
of
logical
plans.
This
sounds
like
a
database
system
because
that's
literally
where
we
took
the
ideas
from
okay
and
you
build
up
a
physical
plan.
A
The
physical
plan
basically
says:
okay,
I'm
going
to
execute
these
rdds
in
a
particular
way
to
basically
give
me
optimal
performance.
The
reason
I
have
a
logical
plan
first
is
because
I'm
saying,
for
example,
oh
maybe
I
should
filter
the
data
before
I
do
the
join.
So
that
way,
there's
less
data
for
me
to
join.
Okay,
that's
why
we
have
this
concept
of
a
logic
plan
versus
physical
plan.
Now
I
can
probably
ask
the
question
you
guys
can
go
for
hours.
A
How
many
times
does
the
statistics
or
what
the
prediction
of
how
large
a
particular
tape
join
table
is
works
out
perfectly.
So
you
probably
know
the
answer
is
not
even
close.
Okay,
so
what's
happening
with
aqe
adaptive,
query
execution
is
that
what's
happening
is
that
when
we
read
the
stats,
when
we
actually
read
the
data,
we
recognize
oh
wait.
The
join
table
is
actually
significantly
smaller.
If
it
is,
then
I
can
do
a
broadcast.
Hash
join
versus
a
shuffle
merge
joint.
A
So
what
ends
up
happening
is
that
in
the
process
we
change
the
join
from
a
standard
shuffle
shuffle
joint
to
a
broadcast
asteroid
that
will
speed
up
the
query
performance.
You
don't
have
to
do
a
thing.
Okay,
so
as
part
of
dbr,
I
figured
which
version
10.1
or
10.2,
but
one
of
those
versions.
Basically,
we
auto
automatically
enabled
aqe,
because
that
was
part
of
spark
3.0.1.
A
I
think,
and
so
because
of
that
that
will
obviously
speed
up
performance
but
assuming
everything's
operating
under
the
spark
3x
line
and
then
the
other
really
cool
thing
which
I'm
going
to
stop
real
soon.
Is
that
also
it's
actually
able
to
handle
skew
joints
really
well,
so,
basically,
aqe
automatically
goes
and
says:
hey
your
data's
skewed
I'll
automatically
sub-partition
the
data.
So
that
way
I
can
go
ahead
and
speed
up
the
career
performance.
So
that
way
you
don't
have
one
or
two
long-running
stragglers.
D
I
mean
yeah,
the
easiest
thing
is
to
lift
and
shift,
and
you
can
basically
just
bring
over
a
lot
of
your
existing
codes.
You
know
so,
if
you
it
depends
on
what
you're
it
depends
consultant
answer
if
you
are
coming
from
something
that
is
heavily
sql
based
and
you
had
a
load
of
sql
stored,
procs
and
that's
how
they
were
doing
all
the
facts
and
dimension
of
things
yeah.
You
can
take
a
lot
of
that
logic
that
you
had
in
store,
prox
and
just
bring
it
over
run.
D
D
We
go
right.
Loop
over
this
list.
Take
these
50
things
run
these
50
notebooks
in
parallel,
so
you're
running
the
one
notebook
with
parameterization,
so
you
can
kind
of
mix
and
blend
so
you're,
keeping
your
business
logic
in
sequel,
where
you
already
had
it
defined,
be
just
wrapping
it
inside
sparked
and
then
kind
of
just
run
it
that's
what
we
end
up
doing
quite
a
lot.
D
D
D
Changing
depends
on
the
object.
I
mean
if
it's
getting
extra
columns,
if
it's
up
scaling
data
types
delta
can
do
that.
That
is
just
a
schema
evolution
of
delta.
If
it
is
okay,
we've
completely
changed
it,
we've
renamed
all
the
columns,
it's
it
is
kind
of
a
drop
and
reload
and
kind
of
reprocess
it
and
you
have
to
take
that
hit,
but
it's
kind
of
just
having
the
processes
built
in
so
you
kind
of
you
can
know
when
those
changes
are
and
you're
kind
of
plugging
it
through
the
process.
D
A
Yeah-
and
one
small
thing
I
would
add,
is
don't
forget:
there's
user
defined
metadata
that
you
can
go
with
your
transactions,
so
common.
Actually
pattern
is
because
you're
putting
your
code
base
in
github
anyways,
you
can
do
the
associated
github
branch,
the
id
for
that
particular
branch
in
which,
when
you
make
to
change
the
code
and
associate
that
with
the
transaction
changes
to
your
table.
So
that
way,
your
table
literally
points
to
the
code
base
that
actually
says.
Okay.
This
is
where
I
made
the
change.
So
that's.
J
One
question
yeah
before
I
ask:
do
we
have
to
leave
the
room
or
we
have
time
to
stay?
I
don't
know
okay
yeah.
Thank
you.
So
I
wanted
to
take
your
advice
on
vacuuming.
Recently,
we've
been
vacuuming
and
we
realized
it
takes
a
lot
of
time
and
our
the
feedback
we
got
from
databricks
was
because
vacuuming
is
doing
that
ls
list
of
all
the
files
and
that's
the
part
that
takes
a
lot
of
time,
and
I
was
wondering
how
we
can
bypass
that
while
doing
vacuuming
or
any
advice
you
have
in
that
action.
A
A
Why
was
because
spark
was
itself
automatically
wanted
to
run
multiple
threads
to
validate
the
data,
but
the
things
if
you're
only
dealing
with
a
small
amount
of
data
in
the
first
place
like
that,
would
work
great
if
you're
talking
about
petabytes
of
data,
but
most
the
time
when
you're
vacuuming
you're
not
trying
to
vacuum
petabytes
of
data.
If
you
do
honestly,
you
have
a
whole
other
problem.
Okay,
so
so
what
ends
up
happening?
Is
you
dock?
The
number
down?
A
So
just
ping
us
on
slack
I'll,
actually
tell
you
exactly
which
which
github
I
just
I
don't
happen
to
have
that
in
my
head
now.
The
second
part
of
it,
though,
is
that,
even
if
pretending
you
knock
down
the
number
there's
these
issues,
don't
forget
that
you
can
actually
spin
up
a
second
smaller
cluster
that
actually
can
run
the
vacuum
in
parallel.
A
Okay,
so
you're,
even
as
you're
doing
inserts
and
updates
and
modifications,
you
can
actually
run
the
vacuums
in
parallel.
This
is
actually
a
very
common
approach
when
it
comes
to
streaming
where,
basically,
you
have
lots
of
little
small
files,
you
don't
care
about
the
historical
context,
because
those
small
files
actually
already
has
it.
So
what
ends
up
happening
is
that
you're
basically
running
the
compaction
and
then
you're
also
writing
the
vacuum
with
a
separate
cluster.
So
that
way,
it's
a
smaller
one.
A
So
it's
not
costing
you
that
much
money
to
basically
go
and
shrink
down
the
the
file
size.
By
doing
that,
you
actually
will
shrink
down
the
number
of
files
that
are
actually
physically
inside
your
storage
system,
which
by
then
very
definition,
even
if
I
have
to
do
a
list,
the
list
is
significantly
faster.
Thank.
J
D
And
that
separate
cluster
I
normally
just
have
running
under
a
databricks
workflow,
not
a
job,
and
then
you
make
a
job
cluster,
so
just
leave
it
nice
and
cheap.
So,
even
if
using
an
interactive
cluster,
which
we
do
quite
a
lot
for
parallelism,
just
just
sort
of
one
cluster
and
then
fire
loads
and
loads
of
things
at
it
in
parallel,
try
and
jam
them
together.
We
can
run
a
separate
thing
as
a
job
cluster
just
to
keep
the
maintenance,
nice
and
cheap.
That's
true,
very
cool
yeah.