►
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon North America 2021 in Los Angeles, CA from October 12-15. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Scaling Apache Spark on Kube to Apple Scale - Amanda Moran & Holden Karau, Apple
Amanda and Holden will explore the customer workloads that easily ported to Apache Spark on Kubernetes, and which ones had more difficulty. The goal of this talk is to help the audience in their journey as either the operators of an Apache Spark-Kubernetes platform or as an end user. Considerations and best practices for end users of an Apache Spark on Kubernetes platform will be discussed. Additional advice for folks migrating from YARN with HDFS to Kubernetes will be included. This talk will include how to effectively deploy the new enhancements of Spark on Kube, like shuffle tracking and graceful decommissioning, as well as when not to use this.
A
A
So,
first
and
foremost,
we're
going
to
tell
you
a
little
bit
about
ourselves,
then
we're
going
to
talk
about
what
is
apache
spark
and
why
it
matters
to
kubernetes
what
is
yarn
and
mesos
spark
standalone
mode
so
get
ready
for
that.
One,
why
you
should
put
spark
on
kubernetes
what
worked?
Well,
while
we
worked
through
this
process
and
what
didn't
work
as
good
best
practices
for
doing
this
kind
of
transformation,
new
features
that
have
been
implemented
and
upstreamed
and
areas
for
improvement.
A
So
who
are
we
so
you'll
see
two
two
of
us
in
this
picture
and
this
collection
of
pictures
work
for
apple,
and
that
is
me
and
holden,
and
you
also
see
our
two
pups
timber
and
jack.
They
support
us
throughout
the
day
as
we
get
our
work
done.
Both
of
us
have
been
and
have
worked
for
apple
for
over
a
year.
A
I
think
holden
is
it's
getting
close
to
two
years
and
both
of
us
have
been
a
part
of
the
apache
spark
community
holden
much
more
so
she's
an
apache
committer
she's
written
books
on
apache
spark
she's
been
with
it
pretty
much
from
the
get-go.
I
came
to
spark
a
little
bit
later,
but
I've
been
involved
with
it
for
quite
a
few
years.
I've
done
quite
a
few
talks
on
spark
and,
and
it's
a
it's,
a
technology
that
I
enjoy.
I
like
teaching
other
people
about
and
using
myself.
A
So
all
right.
So
let's
talk
about
spark
and
what
are
these
things?
So
what
is
spark
so
a
lightning
fast,
unified
analytics
engine
is
one
of
the
taglines
you'll
hear
quite
frequently,
but
apache
spark
is
really
used
to
do
large-scale
data
processing
on
large
data
sets
it's
used
by
data
scientists,
data
engineers,
machine
learning
engineers.
Basically,
anyone
who
is
working
with
large
data
sets
basically
regardless
of
title.
A
So
apache
spark
allows
for
batch
jobs,
streaming
jobs,
the
ability
to
use
spark,
sql,
r,
python,
scala
or
java.
You
can
do
machine
learning
using
spark
and
you
can
also
do
graph
analytics
with
spark
as
well,
so
some
other
taglines
you
may
have
heard
about
spark,
which
is
it's
map
hadoop
mapreduce,
but
with
seven
cups
of
coffee.
A
Of
course,
we
have
a
little
disclaimer
there.
You
know
check
with
your
hardware
vendor
first,
but
I
mean
there
has
been.
I
mean,
there's
been
so
many
numbers
thrown
around
the
years,
but
essentially
you
know
spark
jobs
because
of
their
ability
to
utilize
memory
and
large,
and
do
that
large
processing
anywhere
between
10
to
100
times
faster
than
hadoop,
regardless
of
that's
a
big
range,
but
it's
faster.
A
It
is
a
spark,
is
a
good
way
for
folks
to
learn.
Functional
programming,
which
is
true
since
scala,
is
a
functional,
functional
language.
Me
myself.
I
know
this
much
about
scala
and
I
prefer
to
use
python.
That's
my
language
of
choice
when
using
spark.
So
I
did
not
take
this
opportunity
to
learn
functional
programming
and
it's
a
great
it's
a
great
way
to
use
a
lot
of
compute
resources
right
because
you're
doing
extremely
large
jobs
across
super
large
clusters,
using
a
ton
of
memory
and
cpu.
A
So
let's
talk
about
how
spark
works.
So,
basically,
what
spark
does
the
tl
dr?
Is?
It
spreads
out
compute
across
a
cluster,
most
specifically
a
spark
cluster
right,
there's
two
main
components
when
you
launch
a
spark
job,
the
driver,
which
contains
the
spark
contacts
and
which
works
to
transform
the
user's
code
that
I've
written,
maybe
in
python,
right
splits
it
up
into
tasks
these
bite-sized
chunks
that
can
be
sent
to
be
performed
by
the
executors
and
the
executors
reside
on
each
node.
A
So
from
there
it's
very
easy
to
scale
up
and
scale
down,
depending
on
the
workload,
because
you
can
always
add
more
nodes
and
more
executors,
especially
when
you're
working
in
a
cloud
native
environment
in
the
cloud.
It's
really
easy
to
add
those
extra
resources,
so
apache
spark
has
abstracted
away
from
the
users
any
need
to
have
to
deal
with
orchestrating
data,
processing,
parallelism
or
worrying
about
fault
tolerance.
This
is
all
taking
care
of
them
for
them
so
because
of
the
architecture
and
knowing
that
spark
would
be
spread
across
multiple
nodes
on
very
large
clusters.
A
So
what
is
yarn
yarn
is
yet
another
resource
negotiator,
so
yarn
was
released
in
2012
and
was
a
rewrite
of
mapreduce,
the
mapreduce
engine
from
hadoop
1.0,
so
mr2,
which
was
the
informal
nickname,
even
though
it
really
means
mapper.
It
is
2.0,
which
is
an
application
that
is
actually
managed
by
yarn,
so
it
all
gets
a
little
bit
confusing,
but
it
is
the
de
facto
standard
for
big
data
workloads.
A
So
yarn
supports
a
variety
of
process,
engines
and
applications.
You
can
run
hadoop
and
spark
on
the
same
cluster.
You
can
do
isolation
and
dynamic
allocation
of
resources,
so
yarn
actually
keeps
track
of
the
available
resources.
So
memory
cpu
storage
and
includes
multiple
types
of
scheduling
methods,
so
it
supports
lightweight
isolation
and
it
allows
to
share
local
disk,
and
so
with
all
this
talk
about,
you
know,
resource
allocation
and
managing
resources.
A
A
Not
only
can
you
manage
containers,
but
you
can
also
manage
applications
that
are
not
containerized,
so
it
does
try
to
be
more
than
just
analytics,
like
maybe
yarn,
which
is
being
able
to
manage
hadoop
and
spark
there's
a
private
company.
Now
that
is
dedicated
to
working
on
it
and
they
are
a
bit
more
cube,
kubernetes
focused
than
in
the
past,
and
it
also
allows
for
local
shared
disk
as
well.
A
So
let's
go
into
standalone
mode.
So
are
you
tired
of
doing
your
actual
job,
which
is
writing
java
or
python
and
doing
it
data
analytics,
and
you
want
to
spend
more
time-
writing
shell
scripts
and
managing
your
servers
like
they're,
your
kids
or
your
pets.
Then
this
is
the
mode
for
you,
so
standalone
spark
essentially
a
way
to
create
a
spark
cluster
and
manage
that
yourself.
So
there's
no
support
for
dynamic
resource
allocation
or
resource
management
and
scheduling.
This
is
the
more
painful
option
for
sure,
but
it
is
possible.
A
A
A
I
moved
to
kubernetes
right
so
there's
a
layer
of
reasons
right
well,
first,
this
is
kubecon
right,
so
we
want
to
focus
on
kubernetes,
but
all
jokes
aside
right,
so
one
true
ring
to
rule
them
all
we'll
talk
about
that
more
in
a
second,
you
can
use
your
spare
capacity
for
analytics
and
the
cloud
we're
going
to
talk
about
python
and
what
using
kubernetes
allows
you
to
do
with
python.
We'll
talk
about
security
and
what
that
gives
you
by
using
kubernetes
and,
of
course,
it's
all
about
learning,
new
skills
right.
A
So
just
from
our
quick
review
of
yarn
and
mesos,
it's
easy
to
see
the
benefits
of
needing
a
container
management
platform
when
putting
spark
on
kubernetes,
you
can
add,
spark
and
any
other
types
of
workloads,
so
it
doesn't
have
to
just
be
spark
or
hadoop,
etc.
So
you
can
better
utilize
your
cluster's
resources,
because
you
can
have
all
types
of
workloads
from
across
your
organization
on
one
kubernetes
cluster
so
and
then
just
to
point
this
out
right.
A
So
you
know
many
of
us
in
our
you
know
where
we
work,
we
may
have
just
a
little
bit
of
everything
doing
a
every
type
of
technology
that
there
is
we
have.
We
have
spread
out
across
our
our
companies
right.
So
you
could,
you
know,
have
three
cluster
management
systems
right,
mesos,
yarn,
standalone,
kubernetes,
that's
actually
four
right,
but
managing
those
all!
A
You
know
that
that
leads
to
a
sad
team,
sad
engineers,
because
that's
a
lot
of
you
know
overhead
for
them
to
have
to
try
to
you
know,
switch
between
platforms
and
troubleshooting
et
cetera.
So
of
course
the
solution
would
just
be
turn
or
two
or
three
of
them
off
right.
That'll
just
make
all
the
workloads
better
right.
A
So
it
also
so
like.
I
said
before
you
know
that
you
can
run
different
types
of
workloads
within
that
one
kubernetes
cluster,
so
it
allows
you
to
use
that
spare
capacity.
So
what's
nice
about
kubernetes
again,
is
that
multiple
types
of
applications
and
services
can
be
all
running
on
the
same
cluster.
You
don't
need
a
dedicated
spark
cluster
that
would
be
managed
by
yarn
or
mesos
to
run
your
spark
jobs.
A
A
So,
like
I
kind
of
said,
they're
kind
of
choppy,
the
spark
jobs
you're
doing
can
be
analyzed
any
anything
from
the
various
services
that
are
running
on
the
platform
for
doing
any
other
kind
of
data,
crunching
right
and
so
additionally,
right
when
we're
talking
about
preemption
and
kubernetes,
so
kubernetes
allows
the
ability
to
preempt
workloads
based
on
priority
classes,
which
is
really
powerful,
so
workloads
that
are
less
time
sensitive
than
other
jobs
can
be
rescheduled
when
resource
demand
is
lower,
and
this
is
all
just
can
be
done.
You
know
fairly
automatically.
A
So
next
year,
so
talking
about
containers
and
python
and
security,
so
the
ability
to
easily
move
workloads
from
dev
to
prod
to
add
new
libraries
are
you
know
that
is
because
of
things
like
isolation.
That
cube
has
so
data
engineers
and
scientists
are
always
trying
to
find
new
python
packages
to
add
and
ways.
You
know
new
workloads
that
they
want
to
be
able
to
utilize.
The
latest
versions
of
these
libraries
and
new
versions
of
spark
so
with
a
single
yarn,
mesos
or
standalone
cube
cluster
standalone
spark
cluster.
A
I
should
say
your
apps
will
be
tied
to
using
only
using
one
version
of
spark
or
python.
That's
in
you
know
that's
on
that
cluster,
with
kubernetes
and
containerization
of
sparkler
workloads,
one
data
scientist
can
be
using
the
latest
version
of
spark
so
3.1.1
with
python
version
3.0.
While
your
data
engineer
can
continue
to
use,
spark
2.4,
why
not
with
python
version
2.5
all
along
the
same
cluster,
because
it's
all
containerized
and
isolated.
A
So
also
just
to
add,
when
moving
from
yarn
or
mesos
kubernetes
has
a
lot
of
custom
configurations
that
makes
it
a
bit
more
flexible
than
yarn
per
se
and
last
but
not
least,
learning
a
new
skill
right
for
holden,
and
I
personally
adopting
to
kubernetes
has
allowed
us
to
pick
up
one
more
tool
for
our
tool
belt
and,
of
course,
now
we're
so
much
more
popular
right.
Because
now,
not
only
do
we
know
apache
spark.
We
also
know
all
about
kubernetes.
A
B
Awesome
and
thanks
for
that
introduction,
so
now
I'm
going
to
talk
about
sort
of
the
second
half
of
the
presentation,
namely
what
worked
well
and
where
we
had
rooms
for
growth,
so
small
to
medium-sized
etl
jobs
worked
really
well.
Migrating
them
to
cube
was
relatively
easy.
The
only
thing
that
we
really
had
to
do
was
increase
the
resource
request
to
match
the
reality
of
what
they
were
actually
using
because
yarn
and
mesos
weren't
enforcing
the
resource
limits,
quite
as
strictly
as
cube,
ended
up
enforcing
them.
B
There
were
some
challenges
around
integration
with
the
different
data
sources,
and
that
mostly
comes
back
to
some
networking
configuration
choices
that
were
made,
and
I
think
we
could
make
some
improvements
there.
But
that's
sort
of
just
something
to
keep
in
mind
like
make
sure
that
you
can
easily
access
your
data
sources,
but
it
was
a
relatively
easy
fix.
B
Multi-Language
jobs
are
where
things
started
to
get
a
little
challenging.
So
one
of
the
really
great
things
is
that
dependency
management
improves
substantially
compared
to
yarn
in
the
yarn
world.
All
the
dependencies
had
to
be
managed
by
a
systems
administrator,
whereas
with
running
on
cube,
it
could
be
very
much
more
self-serve.
B
B
Streaming.
Jobs
had
a
lot
of
room
for
growth
and
just
many
areas
of
opportunity
for
investment.
They
just
frankly,
don't
work
very
well
right
now,
and
a
lot
of
that
is
around
the
connection
to
the
different
data
sources.
It
turns
out
that
our
connection
to
the
streaming
data
sources
has
even
more
room
for
growth
than
our
connections
to
the
batch
data
sources.
B
B
Caching
was
never
free
right,
but
now
it
costs
even
more,
and
that's
because
when
we
decommission
an
executor,
we
now
have
to
migrate.
The
cache
data,
and
also
now
for
caching
stuff
to
disk
disk
is
actually
a
metered
resource,
so
be
careful
like
think
about.
Are
you
actually
going
to
use
the
data
twice
if
you're
done
with
it,
tell
spark
you're
done
with
it
right,
especially
for
users
in
no
quick
environments,
the
garbage
collector
isn't
able
to
handle
this
as
well.
B
B
Using
disaggregated
storage
is
super
important.
The
big
thing
is:
there's
no
stay
resident
block
manager,
anymore
and
yeah.
Okay,
data
locality
does
matter,
but
it's
not
enough
to
try
and
co-locate
hdfs,
it's
not
worth
it.
Another
really
interesting
thing
is
that
using
cloud
storage
from
on-prem
doesn't
have
as
much
overhead
as
one
might
think.
B
If
you
structure
your
network
correctly
and
so
initially
in
a
lot
of
situations,
we'd
assumed
that
we
had
to
use
hdfs,
but
after
we
did
some
benchmarks,
it
turns
out
that
it
was
actually
perfectly
reasonable
to
use
cloud
storage
from
on-prem.
So
definitely
like,
don't
just
assume
that
you
need
to
co-locate
hdfs,
take
the
time,
run
the
benchmarks
and
see
if
it's
actually
going
to
be
worth
it
for
you.
B
B
Another
one
is
don't
set
quota
for
the
sake
of
quota
default,
spark
uses
config
maps.
We
ended
up
doing
a
large
rewrite
of
some
code
because
of
a
config
map
quota,
and
you
know
it
turns
out
that
config
maps
weren't
as
expensive
as
we
had
assumed
so
this
is
this
is
very
important
when
you
are
starting
to
set
quota.
B
Definitely
take
the
time
to
see
if
this
is
actually
a
constrained
resource,
or
if
you
know
you
don't
actually
need
to
have
a
quota
here
and
it's
perfectly
fine
to
let
things
run
wild
so,
in
addition
to
sort
of
the
best
stream
practices
for
your
jobs.
Let's
talk
about
what
things
we
changed,
so
we
added
a
new
mechanism
for
dynamic
scaling
and
spark.
I'm
really
excited
about
this.
B
This
is
actually
based
on
a
design
that
I
came
up
with
five
years
ago
and
it
just
didn't
make
sense
back
then,
but
we'll
talk
more
about
it
in
a
little
bit.
If
we
remove
the
commit
fig
map
requirement,
as
we
talked
about,
we
essentially
added
an
alternative
because
of
our
quota
system,
persistent
storage
for
fallback
on
out
of
disk
events,
and
we
were
really
hoping
that
was
gonna
trigger
more
frequently.
B
But
it
turns
out
that
you
know
because
of
how
ephemeral
this
quarter
works,
we
don't
actually
get
the
out-of-disc
events
in
the
same
way.
So
we
we
also
added
some
additional
hooks
inside
of
there
integrations
into
pvcs
some
templating,
the
we
also
gave
users
the
ability
to
explicitly
remove
unneeded
shuffle
files,
and
there
were
a
bunch
of
sort
of
corner
cases
in
sparks
understanding
of
pod
state
that
historically
hadn't
mattered.
B
Only
recently
a
restricted
dynamic
allocation
was
added
where,
if
there
was
no
data
on
an
executor,
we
could
get
rid
of
it,
but
by
adding
graceful
decommissioning.
If
there
is
data
on
an
executor,
we
can
still
get
rid
of
it.
We
just
migrate
the
data
away.
First,
there
are
some
alternatives
proposed
by
other
people
in
the
community
of
adding
a
truly
external
shuffle
service
and
there's
a
few
different
ones
and
we're
not
sure
which
one
is
going
to
land.
B
B
There
are
alternative
proposals
for
for
doing
essentially
more
on
top
of
this,
but
this
is
really
important
because,
with
executor
to
executor
migrations,
we
can
only
scale
down
to
the
point
that
we
still
have
enough
of
mrl
disk
available
for
the
data
and
if
you
have
a
lot
of
data
but
like
let's
say
your
data
scientist
goes
home
at
the
end
of
the
night.
Really,
it
probably
makes
sense
to
go
ahead
and
put
that
data
in
some
kind
of
external
storage.
B
B
It
is
possible
to
turn
on,
but
really
right
now,
as
as
we
saw
from
that
configuration
example,
it
it
sort
of
involves
a
lot
of
fine-tuning
and
playing
with
things.
So
we
haven't
documented
it
because
we
don't
know
what
the
right
settings
are.
Generally,
we
know
what
the
right
settings
are
for
our
cluster,
but
we
haven't
had
enough
other
people
sort
of
play
with
it.
B
So
we
don't
have
good
recommendations
here
yet,
and
so,
if
you
do
want
to
play
with
graceful
decommissioning
and
dynamic
allocation
on
spark
on
cube,
I
would
really
really
appreciate
your
feedback
on
sort
of
what's
working
and
what's
not
working
and
if
you
can
contribute
that
to
the
documentation.
That
would
be
amazing.
B
Another
thing
is
not
all
data
is
equal
right
and
spark.
Has
some
internal
heuristics
around
what
kind
of
data
is
more
likely
to
be
used
or
not?
They're,
not
perfect.
We
could
start
by
applying
those
heuristics
to
block
migrations
or
we
could
try
and
come
up
with
better
heuristics
for
what
kind
of
data
is
worth
migrating.
B
So
more
generally,
spark
on
cube
has
a
lot
of
areas
for
improvement.
Documentation
is
still
there
is
some
documentation
for
spark
on
cube,
but
I
think
this
is
an
area
where
we
can
once
again
improve
a
lot.
Another
one
is
sort
of
q
mechanisms
and
better
understanding
of
the
different
kinds
of
jobs.
That
spark
is
scheduling
and
integration
with
the
cube
scheduler
so
that
we
can
actually
get
faster
spin
up.
B
This
is
something
that
I've
been
working
on
a
little
bit
because
there's
all
kinds
of
reasons
why
things
can
fail,
and
it
can
be
difficult
for
a
user
to
to
get
at
that
information.
It's
not
populated
into
the
spark
web
ui,
and
so
I
think,
finding
ways
to
better
communicate
to
users
what's
going
on
is
really
important,
dynamic,
preemption
priorities.
B
This
is
this
is
really
complicated
in
the
spark
world
and
that's
because
we
definitely
have
this
concept
of
like
pods,
that
are
maybe
more
important,
but
the
problem
is
which
pods
are
more
important.
It's
going
to
change
a
lot
while
our
jobs
are
running
and
even
as
we
change
resource
profiles
or
even
within
the
same
resource
profile
as
data
ages
in
and
out,
and
so
we
don't
have
a
really
good
way
to
communicate
that
to
the
cube
scheduler
right
now,
another
one
that
is
kind
of.
B
Another
one
is
handling
of
jobs,
with
changing
priorities
or
deadlines,
so
we
might
have
a
job
which
is
very
low
priority,
but
it
really
does
need
to
complete
by
the
end
of
the
month,
and
so
we
don't
have
a
good
way
to
express
that
right.
Now,
in
fact,
like
a
you
know,
a
user
would
actually
probably
have
to
cancel
and
reschedule
their
job
at
a
higher
priority
if
it
was
getting
close
to
a
deadline.
B
So
in
conclusion,
spark
on
cube
migrations,
they're,
not
something
that
you
can
just
you
know
set
and
forget,
and
yes,
we
we
can
prevent
spark
more
than
api
servers.
It
has
less
impact
on
on
end
users,
but
there's
still
a
lot
of
work
that
we
can
be
doing
to
make
the
spark
on
cube.
Preemption
experience
better.
B
The
increased
isolation
does
come
with
some
overhead.
I
think
it's
well
worth
it
for
the
benefits
that
we
get
the
security
improvements
and
the
additional
flexibility
around
python
and
native
libraries,
and
also
we
really
like
cake
and
dogs.
You
know
so,
thanks
for
coming,
we
really
appreciate
it.
I
hope
you
are
having
a
wonderful
kubecon
and
please
stay
safe.