►
Description
Notes from the perf lab with fish and joe
Have you ever had poor performing queries in your Spark jobs? Want to learn how some of the most skilled practitioners dissect data performance problems at scale? Join us on the next Data Collab Lab as Franco and Denny compare notes with Chris Fish and Joe Widen about tuning some of the toughest data problems around!
Check out more details here: https://www.meetup.com/data-ai-online/events/274420324/
-----------------
Guests
-----------------
Chris Hoshino-Fish is a Solutions Architect at Databricks. Chris is an active member of the Performance Subject Matter Expert group and a former Principal Consultant focused on Data Engineering, working with several Fortune 500 Databricks customers. Prior to Databricks, Chris worked for an adtech company as a data engineer managing pipelines using Apache Spark for 3.5 years. Chris has a B.A. in Computational Mathematics from University of California, Santa Cruz.
Joe Widen is a Solutions Architect at Databricks. Joe leads the Performance and Delta SME horizontal initiatives along with making customers successful with the Databricks Unified Analytics Platform. Joe has been working with Spark and more generally Hadoop for 5 years, with previous stops at Hortonworks and Capital One.
-----------------
Hosts
-----------------
Franco Patano is a Solutions Architect at Databricks, where he brings over 10 years of industry experience in data engineering and analytics. He has architected, managed, and analyzed data applications both big and small, with open source and proprietary software, utilizing SQL, Python, Scala, Java, and Apache Spark, as well as experimenting with data science. Prior to Databricks, Franco worked as a Data Architect and Analyst in the Commercial Real Estate, Banking, and Education industries for organizations large and small.
Denny Lee is a developer advocate at Databricks, where he works on Delta Lake, Apache Spark, Data Sciences, and Healthcare Life Sciences. He has previously built enterprise DW/BI and big data systems at Microsoft including Azure Cosmos DB, Project Isotope (HDInsight), and SQL Server as well as the Senior Director of Data Sciences Engineering at SAP Concur. Denny holds a Masters in Biomedical Informatics from Oregon Health Sciences University. Databricks is proud to announce that Gartner has named us a Leader in both the 2021 Magic Quadrant for Cloud Database Management Systems and the 2021 Magic Quadrant for Data Science and Machine Learning Platforms. Download the reports here. https://databricks.com/databricks-named-leader-by-gartner
A
You
know
the
biggest
hairiest
problems
that
exist
as
far
as
performance
goes.
Can
you
tell
us
a
little
bit
about
first
off
with
with
joe,
can
you
tell
us
a
little
bit
about
like
your
your
trajectory
here
at
databricks
like
what
you've
been
working
on
and
some
of
the
problems
that
you've
that
you've
seen
out
there.
B
B
B
Over
the
course
of
the
years,
I've
worked
with
people
that
have
you
know
with
reynold
our
cto
with
people
that
worked
on
catalyst,
optimizer
the
team,
that's
building
delta
engine,
the
team
that
built
delta.
So
I've
got
to
work
with
a
lot
of
different
people
and
pick
up
little.
You
know
tips
and
tricks
or
things
they
notice.
I
try
and
remember,
and
that
was
kind
of
the
trajectory
as
far
as
like
hairiest
issues
go.
B
I've
been
in
like
quite
a
few
where
we,
I
shouldn't
say
quite
a
few,
but
they
all
have
a
common
starting
point
and
that
common
starting
point
is
people
will
go
to
stack
overflow.
They'll
watch
spark
session
talks.
They
will
go
to
google
groups,
they'll
go
to
the
spark
apache
user
group,
they
will
go
everywhere
and
they
will
find
configurations
that
they
think
will
make
their
job
faster
and
they
have
no
idea
what
they
do
and
the
hairiest
problems
always
start
there.
B
Did
you
just
over
tune
this
and
you
have
it
everything
perfect?
Or
do
you
kind
of
not
really
know
why
you
did
what
you
did
you
just
did
the
route
of
you
know
every
person,
that's
new
to
a
technology
does
go
through
stack.
Overflow
go
through
the
user
groups,
find
things
that
work
for
other
people
and
hope
it
works
for
you.
So
for
that
one
we
just
literally
removed
everything
and
it
ran
like
three
times
faster.
B
A
D
Totally
yeah,
so
I
became
like
a
junior
data
scientist
at
my
last
company
and
I
joined,
and
I
originally
thought
I
wanted
to
build,
really
fancy
models
and
predict
things.
I
thought
that
would
be
really
cool
and
I
started
actually
doing
it
as
a
job,
and
I
found
I
didn't
like
making
models
that
were
really
bad
all
the
time
and
that
felt
bad
to
make
models
that
were
predicting
things
wrong.
D
All
the
time
and
then
I
found
out
what
I
really
enjoy
is
like
building
the
source
of
truth
so
like
actually
building
and
maintaining
the
data
pipelines
that
analysts
and
data
scientists
will
then
rely
on
that
data
being
able
to
guarantee
some
amount
of
quality
out
of
it
and
ensuring
that
the
pipelines
are
up
and
running
all
the
time.
And
so
then
I
was
kind
of
managing
data
pipelines
for
a
couple
years
at
the
startup,
and
then
we
were
a
customer
of
databricks.
D
D
It
was
it
worked
out
really
well
for
me.
Definitely
you
know
some.
Some
things
are
actually
blessings
in
disguise,
and
then
here
when
I
joined,
I,
we
immediately
started
working
with
one
of
our
healthcare
customers
and
I
got
put
on
a
fairly
long
project
with
them,
maybe
eight
or
ten
months,
and
that
was
where
I
kind
of
got
to.
D
I
had
come
from
an
ad
tech
background,
so
I
I
had
no
domain
knowledge
anymore,
coming
into
this
project
and
kind
of
realized
that
when
you're
coming
in
as
a
smart
data
engineer,
it
almost
doesn't
matter
what
the
what
the
domain
is,
because
data
is
just
data
and
at
the
end
of
the
day
like
what
you
are
doing
to
data
isn't
actually
that
complicated.
There's
only
so
many
things
you
can
do
to
data,
you
can
group
it
in
different
ways.
You
can,
you
know,
collect
it
into
different
data
formats.
D
You
can
but
there's
not.
Actually
you
know
at
a
high
level,
it's
fairly
generic.
You
have
some
data,
you
want
to
make
it
look
different,
and
then
you
want
to
use
that
data
to
tell
a
story,
and
it's
the
story
that
that
actually
matters
much
more
so
than
how
you
got
to
massaging
the
data
into
the
format
you
needed
to.
So
I
got
to
spend
a
bunch
of
time
kind
of
working
at
terabyte
scales,
data
lakes
and
have
a
few
stories.
I
think
my
personal
favorite
is
hdfs.
D
A
Very
nice,
that's
awesome,
so
it's
kind
of
like
thinking
about
performance
issues,
I'm
going
to
be
pulling
up
a
notebook
here
and
kind
of
how
I
want
to
center.
The
next
set
of
questions
around
is
we.
We
talk
a
little
bit
about
delta
lake
while
I'm
pulling
it
up.
I
would
love
to
hear
some
stories.
I
know
chris,
you
were
very
close
and
I
think
you
were
very
close
in
the
initial
build
out
of
delta
lake.
Is
that
correct.
D
What
most
customers
were
running
at
and
would
almost
immediately
find
something
to
break
so
over
the
course
of
a
few
months
they
kind
of
helped
us
very
rapidly
scale
up
to
being
able
to
handle
hundreds
of
terabytes
per
day,
but
while
we're
actually
going
through
it,
it
was
kind
of
like
okay.
Let's
try
this.
Oh
that's
broken
now.
This
works
at
this
scale
when
we
10x
it
doesn't
work
now
so
kind
of
just
taking
things
and
then
10xing
them
and
finding
out
that
they
don't
work
quite
as
well
at
10x
the
scale.
A
A
The
open
source
version
works
on
hdfs,
but
the
databricks
version
works
on
the
cloud
providers
that
we're
on
today.
So
when
we're
thinking
about
these
these
concepts
working
together,
oftentimes
we
have
performance
issues.
So
where
do
you
start?
When
you
have
a
query?
That's
not
performing
very
well!
So
I
have
a
query
in
here
and,
let's
just
say,
I'm
bringing
in
my
data
and
I
I
don't
think
it's
performing
the
way
that
I
think
it
should
be
so
I
have.
A
D
Yeah,
it's
a
great
question
and
I
mean
I
I
personally
think
joe
is
even
better
at
this
than
me.
I
always
start
with
the
stages
tab
of
the
spark
ui
the
stages
tab
is
where
you
get
the
the
most
amount
of
information
in
a
quick
way.
You
get
the
the
summary
of
each
stage.
You
get
the
duration
that
it's
taking
and
you
also
get
to
see
the
amount
of
data,
that's
processing.
D
So
if
you
expand
the
completed
stages
yeah,
so
here
right,
you
get
to
see
the
duration
of
each
stage,
how
many
tasks
there
were
and
then
inputs,
outputs
and
shuffle
sizes
shuffle
file
sizes.
So
this
is
where
you
can
start
to
see.
You
know
this
is
where
you
kind
of
develop
your
intuition
a
little
bit
and
say:
okay,
based
on
the
amount
of
input
data
that
we
had
and
the
number
of
tasks
that
we
had.
Is
that
a
slow
stage
or
is
that
a
fast
stage
or
a
normal
stage?
D
Not
necessarily
right,
I
think,
shuffle
is
slow,
definitely
and-
and
you
know
just
for
anybody
out
there-
who's
wondering
shuffling
is
when
you
need
to
exchange
data
across
executors
right.
So
we
have
a
cluster
of
machines
and
there
are
some
operations
where
we're
required
to
send
data
out
across
the
network
and
exchange
data
across
machines,
and
so
this
might
be
when
you're
aggregating
data,
each
executor
is
going
to
do
local
partial
aggregation
if
possible.
D
But
then
it's
going
to
eventually
need
to
send
all
the
results
to
a
single
machine
to
get
you
that
final
answer,
and
so
shuffle
is
required.
It's
not
bad
necessarily,
but
we
should
always
be
thinking
that
shuffle
requires
network
transfer
and
network
transfer
is
always
going
to
be
slower
than
processing
the
data
locally.
On
the
same
machine.
D
D
Those
nodes
are
virtual
nodes
and
some
of
them
might
actually
end
up
physically
on
the
same
machine,
but
if,
instead
of
adding
just
more
nodes,
if
you
switch
over
to
larger
nodes
and
instead
of
just
adding
eight
times
more
nodes,
if
you
switch
to
the
eight
x
larger
nodes,
that's
where
you
can
start
to
see.
You
know
more
local
operations.
So
now
you
have
more
cores
that
are
located
on
the
same
executor,
and
so,
if
these
tasks
need
to
shuffle
data,
they
don't
necessarily
have
to
shuffle
it
from
one
executor
to
another.
D
What's
called
the
map
side
of
the
shuffle
and
then
the
reduced
side
of
the
shuffle,
and
so
on
the
map
side,
each
executor
will
write
out
the
data
that
it
has
partially
aggregated
or,
if
you're,
joining
partially
keyed,
and
it
will
write
that
data
out
to
the
local
disk
of
the
executor
as
shuffle
files
and
then
on.
The
reduce
side
of
the
executor.
D
A
Interesting,
so
does
it
work
on
so
joe?
I
know
that
that
we're
talking
bigger
nodes
here,
but
on
the
flip
side
I
know
that
we've
actually
seen
the
other
side
of
this
actually
show
better
performance
where
you
know
maybe
not
as
large
nodes,
but
not
smart,
not
like
really
small
nodes
either.
I
know
you
have
a
couple.
We
have
a
couple
theories
around
this.
You
may
talk
about
that.
A
little.
B
Yeah,
so
I
don't
want
to
go
directly
into
that
theory,
because
if
it's
wrong,
I
don't
want
to
look
dumb,
but
basically
what
we've
found
is
a
like
when
you
think
about
it.
What
chris
says
makes
sense
like
if
you
could
have
10
machines
with
10
cores
or
one
machine
with
100
cards.
Well,
there's
no
network
that
needs
to
move
because
everything's
on
the
same
machine
and
in
theory
that
that
all
makes
sense.
But
in
practice
what
we
tend
to
see
is
actually
the
medium-sized
machines
do
best.
B
I'm
sure
that's
you
know
not
going
to
cover
every
edge
case,
but
mostly
that's
the
case,
and
one
thing
I
would
just
kind
of
add
to
chris's
comment.
Shuffle
is
shuffle
there's
a
lot
of
times.
You
can't
get
around
it,
but
what
you
can
do
is
look
for
strategies
to
reduce
it,
and
the
way
you
can
reduce
it
is
by
doing
more
broadcast
joins
by
reordering
your
join.
B
So
you
join
the
smaller
tables
first
and
then
then,
the
bigger
table,
the
bigger
table,
the
bigger
table
and
the
whole
goal
is
just
kind
of
think
about.
How
can
we
move
the
least
amount
of
data?
So
if
you
join
the
two
smallest
tables
to
the
next
smallest
table
that
table's
slowly
getting
bigger,
but
if
you
think
about,
if
you
join
the
biggest
table
to
the
biggest
table,
you
have
to
propagate
that
entire
data
set
down
your
dag.
A
A
Essentially,
what
we're
saying
is
that
when
we're
talking
about
cluster
sizing,
really
we're
still
figuring
out
like
but
we'll
we're
still
experimenting
with
different
sizes,
and
it
all
depends
kind
of
on
your
data
volumes
and
what
you're
doing
this
is
my
theory.
I
think
that
I
think
you're
you're
on
point.
I
think
that
it's
data
volumes
and
then
it's
complexity
of
all
the
transformations.
So
are
these
compute
heavy?
Are
they
io
heavy
like
there's
a
lot
of
data
coming
in
and
out?
Is
it
processor
heavy,
so
is
the?
Is
it
compute
like?
A
Are
we
compute
strain
some
things
that
I've
heard
in
the
recent
past
is
like?
Where
are
we
constrained?
Does
it
compute?
Is
it?
Is
it
cpu
side?
Is
it
memory
side?
Is
it
network?
Are
we
I
o
bound
like
where?
Where
is
the
bottleneck?
Where
would
you
so
I
I
I
know
I
have
like
mine,
but
where?
Where
would
you
go
to
kind
of
if
you
had,
if
you
could
just
see
like
a
big
picture
of
where
to
quickly
spot,
where
we
could
be
bound.
B
B
Yeah,
so
this
is
going
to
tell
us
what
our
our
cluster
is
doing.
You
have
a
very
underutilized
cluster,
so
it's
going
to
be
hard
to
to
exactly
show,
but
basically
you
can
see
all
of
or
almost
all
of
this
from
right
here.
So
the
first
part
we
usually
look
at
is
I
o
bound,
so
I
o
bounds
when
you're
reading
as
fast
as
you
can
from
your
storage
system.
So
what
you'll
see
is
on
the
right?
There
you'll
see
your
cluster
network
level.
B
B
If
it's
highly
compressed,
then
you're
going
to
be
cpu
bound,
but
if
it's
not
really
highly
compressed
you're
just
going
to
be
network
bound
ios,
typically
well,
they
only
happen
on
reads
and
writes
it's
pretty
intuitive.
So
that's
kind
of
the
first
I
o
bound.
The
next
part
here
is
cpu
bound.
It's
pretty
obvious
that
when
you're
circling
right
there
you'll,
basically
max
it
out
around
80
or
90
percent,
it's
pretty
common
to
be
cpu
bound
when
you're
decompressing,
large
parquet
files,
large
gzip
files,
anything
that's
compressed.
B
Some
operations
are
more
cpu
intensive
than
others.
It'll
be
very
apparent
because
you'll
be
using
like
90
or
95
of
your
cluster
and
that's
what
I
would
consider
cpu
bound
memory
bound
is
a
little
bit
tougher
to
diagnose
so
there's
memory
being
used
by
the
cache
and
by
your
jvm
and
your
different
regions
in
your
jvm,
and
this
tells
you
approximately
how
much
you're
using
so
if
franco,
if
you
were
memory
bound
or
if
we
thought
memory
might
be
an
issue,
you
would
see
this.
Basically,
the
blue.
B
B
B
B
A
Got
it
so,
if
so,
spark
is
in
a
little
bit
of
background
here,
spark
is
in
memory,
cluster
compute
right,
so
everything
goes
to
memory
to
to
compute
because
it's
faster
right
than
going
back
to
disk.
So
if,
if
the
memory
is,
if
you're
out
of
memory,
so
if
you've
exhausted
all
the
memory
on
the
box,
then
it
will
then
go
to
the
disk.
A
So
it'll
write
the
data
to
the
disk
instead
of
the
memory,
because
there's
no
more
memory
so
essentially
spill
means
that
your
memory
is
full
and
now
it's
spilling
over
to
the
disk
so
spilling
to
this
costs
you
time
right,
because
disk
is
much
slower
than
memory.
So
we
we
want
to
reduce
the
amount
of
spill.
What
are
some
strategies
to
reduce
the
amount
of
spill.
B
B
B
Yes,
that's
a
config.
I
would
generally
never
recommend
reducing
the
cores
unless
you
really
know
what
you're
doing.
I
guess,
for
example,
that
customer
I'm
working
with
on
the
other
side
of
the
world.
We
were
running
on
ds15s
on
azure.
It
has
20
cores
and
we
were
seeing
some
spill
and
we'd
already
moved
our
sql
partitions
to
20
000.
But
by
going
from
20
cores
to
18
cores,
we
were
able
to
actually
get
the
run
time
from
basically
an
hour
to
54
minutes
and
they
wanted
to
break
an
hour.
So
it
wasn't
intuitive.
C
Oh
sorry,
joe,
I
was
just
curious-
was
it
partially
because
maybe
it
had
to
cross
sockets
as
well?
Maybe
the
the
fact
that
it
was
actually
dealing
with
the
cache
and
the
s
and
when
you're
going
between
you
have
that
too
many
cores
on
that
particular
machine.
Perhaps
that
was
coming
into
play.
At
least
it
used
to
happen
to
me
all
the
time
in
the
past,
but
I
don't
know
if
that's
in
play
any
longer.
B
Yeah,
so
that's
part
of
our
long-standing
kind
of
theory
about
how
the
really
big
machines
with
aggregations.
Don't
work
really
well
spark
is
what
we
call
no
to
wear,
but
it's
not
like
core
or
cash
aware.
So
when
you
have
like
one
core
reaching
into
another
course
the
l2
cache
you
get
the
data,
then
you
know
you
slow
down
right.
It's
a
theory.
C
C
We
definitely
noticed
that
when
we
were
trying
to
with
sql
server
because
it
was
going
across
the
cross
bridges
between
the
sockets
right,
what
ended
up
happening
is
that
because
they
actually
hit
the
cross
bridges
massive
drop
in
performance,
so
we
actually
had
to
allocate
allocate
memory
to
specific
cores
in
order
to
be
able
to
prevent
the
hitting
the
l2
and
l3
caches
just
like
you're
talking
about.
So
I'm
wondering
if
we're
literally
running
into
the
same
problem
all
over
again
now.
C
A
C
Absolutely
yeah
yeah.
We
were
mucking
around
with
it,
trying
to
reconfigure
the
assignment
groups,
any
large
systems,
basically
what
we
realized
that,
even
before
we
had
the
problem
of
like
when
we're
doing
sharding
that
that's
exactly
it,
that
would
literally
were
in
essence,
sharding
within
the
same
machine
because
they
became
large
enough
where
basically,
the
pneuma
94
memory
allocation
just
actually
had
a
massively
large
impact
in
performance.
A
That's
awesome
so
kind
of
just
to
talk
about
the
memory
to
core
calculation
that
you
were
talking
about.
If
we
look
here
on
the
drop
down
that
this
is
what
you're
talking
about,
there's
four
cores
sharing
30
and
a
half
gigs
of
memory
and
each
instance
type
whether
whatever
cloud
you're
on
right
is
this.
What
you're
talking
about
are
we
talking
actually
like
the
closer
memory
to
the
processor.
A
B
There's
gonna
be
times
where
the
general
processing
of
like
an
m4
is
fine,
and
I
know
we
give
you
all
of
these
options,
but
usually
using
the
c5
8x,
the
c5d9x
or
the
i3.
4X
is
good
enough.
I
mean
you
can
try
everything
you
want,
but
at
the
end
of
the
day
those
three
will
probably
be
best.
So
don't
let
you
know
don't
get
too
hung
up
here.
A
Yeah,
I
I
agree.
The
the
one
thing
I
want
to
say
about
about
choosing
the
right.
A
lot
of
times
is
choosing
the
right
infrastructure
for
the
right
job.
I
know
fish
you've
got
some
some
tips
here,
based
upon
some
things,
you've
seen
and
especially
I've
actually
done
some
testing
here.
I
really
like
the
c5ds
c5ds
actually
include
locally
attached
fast
disk,
and
you
can
enable
delta
cache
on
these
cluster
types.
So
you
gain
the
benefits
of
compute
optimized,
along
with
the
locally
attached
fast
disk
for
delta
cache.
A
D
Yeah,
definitely,
I
think,
we've
both
joe
and
I
have
worked
with
a
bunch
of
customers
who
have
done
some
pretty
serious
testing
of
workloads,
and
I
mean,
if
you're
a
data
engineer.
I
think
this
is
one
of
the
most
fun
things
that
you
get
to
do
perhaps
and
fun
in
a
boring
way,
but
just
getting
to
run
a
workload
that
you
know.
D
C
That
I
was
about
to
ask
you
that
question
because
it
looks
like
c5s
are
also
using
non-volatile
memory
as
their
local
storage
right,
and
so
it
seems
to
me
that
that
so
I
think,
there's
an
implication
of
basically,
even
though
spark,
if
it
does
spill,
the
ducts
has
a
tendency
to
be
the
random
iops
right,
because
it's
ssds
it
doesn't.
I
wouldn't
say
it
doesn't
matter,
but
the
performance
is
not
that
nearly
as
slow
just
because
it
is
on
ssds.
So
I
just
want
to
add
that
factory.
A
D
Interesting
then,
the
other
thing
I
want
to
point
out,
while
we're
here
actually
is
the
network
bandwidth
column.
So
this
is
something
a
lot
of
people
don't
realize,
but
you
know
under
the
hood
right.
These
are
all
virtual
machines.
So
when
you,
when
you
request
a
c5
x,
large
they're,
giving
you
1
24th
of
a
c5
24x
large,
like
a
c524x
large,
is
the
actual
whole
machine
in
a
server
rack.
D
D
But
when
you
request
the
smaller
instance
types
you're
getting
a
fraction
of
that-
and
what
you
can
see
is
that,
as
you
go
up
the
instant
size
sizing,
there
is
a
break
point
where
you're
actually
guaranteed
now
a
certain
amount
of
bandwidth-
and
so
here
here.
The
break
point
for
me
is
the
9x
larges,
because
under
that
you're
only
guaranteed
up
to
10
gigabits
per
second,
and
they
will
throttle
your
bandwidth
if
there
are
other
machines
on
that
same
rack.
D
Using
a
lot
of
bandwidth,
but
if
you
go
up
to
the
9x,
larges
you're
now
guaranteed
a
static
10
gigabits
per
second,
as
the
minimum
throughput
that
you're
going
to
get.
And
so,
if
you
have,
these
workloads
like
as
joe
is
mentioning
most
etl
workloads
are
actually
network
banned
with
network
bottleneck
and
so
by.
Moving
up
to
these
larger
instance,
types
where
you're
now
guaranteed
a
certain
amount
of
throughput,
you
can
actually
end
up
getting
better
rights
going
through
to
to
especially.
A
That's
that's
great
and
that's
exactly
what
I
wanted
to
get
to
by
showing
these.
So
thank
you
for
pointing
that
out.
So
when
we
were
talking
about
being
network-bound
or
being
memory
bound,
and
we
had
kind
of
the
core
to
memory,
you
know
conversation
so
when
we're
talking
about
being
network
bound
or
being
compute
bound,
so
there's
multiple
ways:
it's
not
just
you
know
configuring
spark.
Sometimes
we
need
to
think
about
the
virtualized
infrastructure
that
we're
spinning
up
in
the
cloud
underneath
the
cupboards
and
chris
pointed
out
a
very
good
point.
A
A
Like
chris
mentioned,
you
have
a
very
large
rack
right
and
then
they
they
virtualize
the
servers
and
they
they
slice
everything
up
right,
so
yeah
up
to
10,
you're,
basically
sharing
hardware
right
with
other
people,
so
you're
sharing
the
nick
the
network
card.
Whereas
if
you,
if
you
move
up
you
kind
of
get
guaranteed,
as
you
see,
if
you
go
up
even
further,
you
get
guaranteed
25
gigabits
per
second,
and
I
know
that
there
are
some
exotic
it
correct
me.
D
No,
I
think,
some
of
the
newest
instance
types
amazon
is
starting
to
build.
You
know
their
own
hardware
accelerators
on
top
of
these
server
racks,
so
I
think
there
definitely
is,
but
those
kinds
of
things
are
usually
you
know
you
have
to
think
about
the
hierarchy
of
needs
in
the
cloud
like
those
things
are:
first,
usually
targeted
for
web
facing
stuff
right,
so
like
web
facing
stuff
still
has
the
highest
network
needs,
but
definitely
sort
of,
as
s3
become
an
object.
D
Storage
in
general
that,
like
you
said
this
is
not
unique
to
aws.
Azure
has
the
same
details
for
their
vms
as
more
and
more
people
are
using
object,
storage
for
analytics
right,
the
the
network,
throughput
of
the
object,
storage
becomes
more
and
more
obvious
to
the
users,
and
so
I
think
the
clouds
are
very
much
working
on
this
to
to
accelerate
these
speeds
inside
of
their
inside
their
data
centers
and
across
data
centers.
A
Yeah
there's
a
similar.
I
can't
I'm
not
finding
it
here,
but
there's
a
similar
chart
on
the
azure
side,
where
it's
a
very
similar
experience
where
you
have
different
compute
sizes
and
then
they
kind
of
go
up
here.
It
is
this
is
the
bs
series
and
it's
the
same
concept
right
you've
got,
it
doesn't
look
like
they
have
nick
on
here.
D
A
A
So
I
just
wanted
to
spend
a
little
bit
time
talking
about
virtualized
hardware
when
we're
talking
about
kind
of
diagnosing
performance
problems.
So
when
you're
in
ganglia-
and
you
kind
of,
can
see
that
you're
bound
by
one
of
these
three
one
of
these
three
right
you're,
either
going
to
have
memory,
cpu
or
network,
you
know,
there's
there's
more
than
one
way
to
kind
of
dissect
and
understand
and
solve
that
problem.
You
know,
spark
configs
is
one
thing,
but
there's
only
so
much
you
can
do
there
right.
A
Really,
let's
take
a
look
at
the
virtualized
hardware
under
the
covers.
Let's
take
a
look
at
what
resources
I
have
available
and
sometimes
choosing
a
different
instance
type
will
get
you
better
performance
and
in
the
cloud,
if
you
pay
for
infrastructure
over
time,
and
and
if
you
boot,
a
c5
d12
xl-
and
it
finishes
in
let's
say
a
fourth
of
the
time
that
a
c5
d2xl
does
and
it's
a
little
bit
more
pricey.
You
could
actually
be
cheaper
by
going
to
the
bigger
instance
type.
D
A
C
Yeah,
we
probably
have
about
four
four
to
five
more
minutes,
so
we
can
probably
go
a
little
bit
more
before
we
do
it,
but
I
did
want
to
call
out
actually
we
did
have
some
questions
from
the
audience
specifically
around,
like
which
one
which
instance
types
do
you
want
to
use
for.
You
know
etl
versus
machine
learning
versus
aggregate,
and
I
I
was
basically
sort
of
answering
the
questions
on
on
live.
Basically
saying
the
answer
is
always.
It
depends
right.
I
mean
because
the
reality
is
your
etl
process
may
be
streaming
related.
C
So
then,
yes,
it's
perfectly
fine
to
be
more
core
slash
memory,
heavy
versus,
maybe
aggregate
heavy.
Well,
then,
maybe
you
don't
want
so
many
different
notes
right
you.
So
that
way
you
can
reduce,
shuffling
and
so
forth
and
so
forth,
and
even
what
I
just
said
isn't
necessarily
correct
right
there,
I'm
more
than
sure
joe
and
chris
and
and
everybody
else
for
that
matter
can
pull
up
an
an
example
where
what
I
just
said
is
wrong.
Okay.
C
B
B
A
That's
great,
that's
that's
the
hardest
part
for
for
someone
like
me
who
was
learning
the
stuff
on
the
fly
a
couple
years
ago.
The
hardest
part
is
like:
where
do
I
start
right,
and
I
think
that
I
think
we've
given
some
really
good
tips
today
on
where
to
start,
I
had
a
bunch
of
other
questions.
I
think
we
got
some
covered
some
great
stuff
today.
Danny
do
we
want
to
take
another.
Is
there
another
good
question
that
we
got
on
the
q
a
out
in
the
wild?
C
D
Okay,
yeah
yeah,
I
think
I'll
tell
the
I'll
retail
the
name
node
story,
the
we
we.
Basically
we
had
this
poc
going
on
where
we
needed
to.
We
wanted
to
do
a
bake
off
on
two
on
an
80
terabyte
data
set
and
a
bake
off
between
a
couple
different
data
formats
and
a
couple
different
engines
and
at
80
terabytes.
D
It's
it's
actually,
as
you
might
imagine,
it's
pretty
slow
to
actually
alter
the
data
format,
and
so
this
was
kind
of
one
of
those
things
that
really
just
made
it
obvious
how
much
delta
has
changed
the
way
we
deal
with
data
engineering
like
one
of
the
formats
being
baked
off,
was
orc
and
and
hive
acid
tables
and
so
etling.
The
data
into
that
format
took
several
days
and
when
there
were
failures
you
had
to
go
in
and
do
a
lot
of
manual
cleanup.
D
I
just
streamed
it
in
and
said
auto
optimize
and
optimized
rights
on
and
just
kind
of
left
it
alone
for
six
hours,
and
then
it
was
done,
and
you
know
there
may
have
been
some
intermittent
failures
in
the
middle
of
it,
but
the
stream
just
recovered
and
restarted
and
delta,
because
it
it
automatically
deals
with
those
we
never
had
to
worry
about
it.
On
the
read
side,
just
kind
of
left
things
along.
B
Brief,
so
one
of
the
bigger
ones
I've
worked
on
recently
was
a
daily
etl
job
that
was
joining
like
you
know:
55
60,
terabytes
of
data
to
20
other
tables,
and
we
ran
into
issues
not
really
related
to
spark,
but
just
like
running
spark
at
that
scale
in
the
cloud.
So
things
like
the
different
like
s3
goes
on
aws,
so
s3
throttled
us
pretty
good
requesting
a
cluster.
That
big
was
a
little
kind
of
shaky
using
spot
instances.
B
We
flooded
the
spot
market
so
that
one
was
probably
one
of
the
bigger
ones
and
it
led.
I
mean
that's
kind
of
how
we
know
some
of
these
hardware.
Things
is
when
you
get
to
a
certain
size
like
it's,
not
spark,
that's
the
problem,
it's
the
hardware
in
the
cloud
and
everything
else
that
goes
and
interacts
with
it,
so
that
was
probably
one
of
the
bigger
headaches,
but
more
fun
ones
and
things.
I
learned
a
lot
from.
A
That's
awesome.
I
just
want
to
touch
on
a
couple
of
things
when
chris
was
talking
about
the
fault,
tolerance
of
delta
and
streaming
just
to
provide
a
little
bit
of
color
there.
The
structure
streaming
has
the
checkpoint
directory,
which
keeps
track
of
everything
all
the
files
that
it's
ingesting
delta
has
the
transaction
log.
These
two
things
work
together.
So
if
something
happens,
midstream
fails,
the
new
cluster
can
boot
up
and
figure
out
where
the
heck
it
landed
and
it
basically
picks
up
from
where
it
left
off
and
it
continues
processing
did.
I
did.
A
A
C
C
So
for
starters,
I
want
to
say
thank
you
very
much
to
fish
and
joe
for
this
awesome
session
with
franco
and
myself
really
good
info,
like
always
for
those
of
you
wondering
the
notebook
that
franco
was
playing
with
as
well
as
this
recording
both
on
linkedin
live
and
youtube.
Well,
they're
already
on
linkedin
live
and
youtube,
so
you
have
the
recordings
the
notebooks
will
put
in
github.comtechtalks,
it's
the
usual
location
for
these
series.
So
that
way
you
can
go
play
with
the
notebook
as
well.
So
again,
thanks
very
much
these
gentlemen.
C
Let's
go
ahead,
we
want
to
do
some
quick
call
outs
for
starters,
for
example
the
data
and
ai
summit
europe,
that's
actually
next
week,
okay
next
week
in
gmt
time
zones,
formerly
known
as
the
spark
and
ai
summit,
so
we
have
some
awesome
names.
We
have
some
really
cool
folks.
In
addition
to
mae
jemison
who's,
this
awesome
nasa
former
nasa
astronaut,
which
of
course
blows
to
my
heart,
there's
also
malcolm
gladwell,
which
he's
amazing.
C
If
you
get
nothing
from
this,
read
out,
read
outliers
his
book
and
read
and
listen
to
his
podcast
for
finnish
revisionist
history
seriously
good
stuff.
We
have
multiple
meetups,
we
have
ama
sessions
and
we
have
training.
There's
a
25
off
training
here,
dais
20
meetups,
you
use
that
you
get
25
off
so
wanna
call
that
out.
C
If
you're
interested
in
presenting
okay
submit
your
project
for
the
data
in
ai
summit,
europe
for
community
lightning
talks,
they're
gonna
actually
happen
in
european
time
zones,
so
be
aware
of
that
fact
they
will
be
live.
They
will
be
fast,
submit
your
project
just
go
to
data
and
ai
summit
europe
you're
going
to
be
able
to
see
the
link.
C
There
click
on
it,
the
first
10
to
submit
and
their
projects
will
receive
also
free
vip
passes,
which
include
exclusive
amas
on
delta
and
photon,
which,
by
the
way
the
gentlemen
here
on
this
on.
This
call
are
going
to
also
be
on
some
of
those
ama
sessions
as
well.
Okay
for
the
meetups,
we
have
the
databricks
university
alliance
meetup
just
on
the
17th.
C
With
with
these
fine
people,
we
have
olivier
gerard
from
nba
esg.
We
have
kyle
hamilton
from
uc
berkeley
and
we
have
linus
wallin
from
yeonping
yonkoping
university.
I
apologize
if
I
said
that
wrong
and,
of
course,
rob
reed
from
databricks
who's,
our
university
alliance
manager.
He
will
be
there
to
go
ahead
and
emcee
that
session.
Okay,
so
you
want
to
know
a
little
bit
more
about
how
to
use
databricks
in
your
university
boom.
This
is
the
meetup
to
attend.
C
We
also
have
a
data
and
ai
online
meetup
on
apache
spark,
3.0,
internals
and
building
structure
streaming,
microservices
architectures
with
scott
haynes
from
twilio
and
jastic
lozowski
who's,
an
independent
consultant
but
writes
the
very
popular
inter
spark.
Internals
ebook,
okay,
it'll,
be
hosted
by
my
jules
and
myself
so
they'll
be
on
wednesday
and
then
on
the
following
day.
C
We'll
actually
have
another
dnai
online
meetup
for
mlflow
delta
lake
and
lake
house
use
cases
so
with
andre
massarovic
and
oliver
kornig
from
databricks
on
ml
flow
and
lake
house
scenarios
with
delta
lake,
of
course,
and
again
all
hosted
by
jules
and
myself.
So,
as
you
can
tell
lots
of
fun
stuff
next
week
so
and
even
though
we've
scheduled
them
primary
for
gmt
times
for
those
in
the
us,
because
they're
late
enough
gmt,
they
actually
should
be
somewhat
comfortable
for
those
in
the
american
time
zones
as
well.
C
Okay
and
final
little
spiel,
don't
forget
to
check
out
databrew.
This
is
a
vidcast
podcast
series
by
myself
and
brookwood.
Nick
brook
is
the
real
star
of
the
show
we're
gonna,
be
we
discuss
with
external
panelists
on
data
in
the
data
and
ai
community
on
a
lot
of
really
cool
topics.
For
example,
the
first
session
is
with
barry
devlin,
susan
o'connell
and
donald
farmer.
C
If
you're
familiar,
you
may
be
familiar
with
some
of
those
names,
especially
for
coming
from
the
data
warehousing
world
from
data
warehousing
to
data
lakes
in
about
40
minutes,
it's
actually
more
like
44,
but
it
whatever
so
anyways.
Definitely
check
it
out.
Just
go
to
davrix.com,
discover
data
dash
brew
you'll,
be
able
to
hear
the
series
we
actually
will
be
pushing
out
podcasts
about
every
two
to
three
weeks.
C
The
next
one
actually
would
be
with
our
ceo,
a
ceo
ali
ghazi,
so
perfectly
good
stuff
and
off
fun
stuff
to
be
listening
to
okay
and
don't
forget
also
the
third
one
will
also
be
with
michael
armbrust,
so
we
get
all
geeky
as
well.
So
that's
it
for
this
field.
For
today
again,
I
want
to
thank
chris
and
joe
for
their
for
presenting
the
their
session
franco,
like
always
for
going
ahead
and
running
the
show.
This
is
awesome.
A
C
I
completely
disagree,
yes
all
right!
Well,
I
think
that's
it
for
today.
Thank
you
very
much
everybody
for
attending
today's
session.
We
apologize
for
not
necessarily
be
able
to
answer
all
questions,
but
we'll
we'll
be
online
again,
like
usual,
and
definitely
look
for
us
next
week
at
daytona
summit,
our
ugly
mugs
will
be
up
there.
Okay.