►
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Apache Spark on Kubernetes: A Technical Deep Dive - Yinan Li, Google
Apache Spark is currently the most popular open-source large-scale data processing framework. Previously, users could run Spark applications on standalone, Yarn, and Mesos clusters. In the Spark 2.3.0 release, Kubernetes became a new scheduler backend for Spark. This new scheduler backend enables Spark applications to run natively on Kubernetes by leveraging the Kubernetes scheduler for scheduling and running Spark drivers and executors. In this talk, we will give a deep dive into the technical details of the Kubernetes scheduler backend and explore all the exciting new things that this native Kubernetes integration brings to Apache Spark. We will also go over the roadmap and features that the Kubernetes community has planned for the scheduler backend over the next several releases of Spark.
To learn more click here: https://sched.co/FuJS
A
Okay,
hello,
everyone,
so
my
name
is
Tina
and
I'm.
Gonna
talk
about
party
smart
in
communities,
so
basically
I'm
going
to
give
a
technical,
deep
dive
into
like
I
know
some
internal
of
this
project
so
for
the
rest
of
the
presentation.
I'm
gonna
use
English,
actually
so
because
for
some
of
the
terms
really
hard
to
translate
them
into
Chinese
a
back
and
forth
so
I'm
gonna
stick
to
stick
to
English
for
the
rest
of
the
presentation.
A
The
project
so
falling
that
I
will
give
you
know
to
technical,
deep
dive
into
some
of
the
internals
of
how
everything
works
together.
So
then,
we'll
talk
about
some
of
the
existing
features
that
we
have
already
made
into
spark
and
okay
book
overview
of
you
know
some
of
the
future
roadmap.
Like
you
know,
so
what
do
you
can
expect
from
the
the
community?
So
what
kind
of
new
feature
you
will
see?
You
know
upcoming
smart
radius.
A
C
A
Equation
so
like
there's,
you
know
a
lot
of
reasons
why
people
want
to
run
spark
natively
on
communities,
but
I
think
these
are
the
main
reasons
behind
the
project.
So
one
of
the
main
reasons
like
you
know
so
the
rapidly
growing
ecosystem
around
docker,
container
and
kubernetes,
so
people
might
have
already
been
running.
Like
you
know,
a
lot
of
these
tuning
for
their
online
serving
workloads
were
stateless.
We're
closed.
So
naturally
they
want
to
run
all
these
tooling.
You
know
also
to
a
four
patch
kind
of
workloads
or
data
processing
workloads.
A
So
they
wanted,
like
you,
know,
leverage
these
rich
tooling,
also
for
their
batch
workloads.
So
another
like
nurse,
are
an
increasing
trend
of,
like
you
know,
people
adopting
multi
cloud
or
hybrid
cloud
environment,
so
you
might
have
like
you
know
you
might
use
services
from
more
than
one
cloud
providers
like
and
you
might
run
some
services
on
Google
or
some
part
of
the
services
on
yeah,
for
example,
Amazon.
So
also
some
companies
actually
day
to
hybrid
set
up
so
they'd
run
part
of
their
workloads
on
in
a
cloud
in
the
meanwhile.
A
They
also
run
some
of
their
workouts
on
their
own
prime
datacenters.
So
in
that
case,
like
a
portability
is
the
key
like
I
know,
so
you
cannot
you
want
to
achieve
portability
by.
Basically,
you
don't
need
to
do
anything.
Then
you
can
move
your
workloads
from
one
environment
to
the
other.
So
while
kubernetes
is
the
key
to
achieve
you
know
portability,
so
naturally
we
want
spark
to
be
in
a
to
be
able
to
natively
run
on
kubernetes.
So
last
but
not
least,
we
see
a
trend
like
people
wanted
to
consolidate
their
hardware
resources.
A
So
traditionally
people
run
big
data
workloads
on
things
like
yarn
or
missiles.
Well,
we're
seeing
recently
like
people
wanted
to
move
all
these
data
processing
workloads
also
to
communities
well
for
the
reason
like
in
they're,
already
operating
coordinators
class.
Therefore,
there
are
online
serving
workloads,
so
they
don't
want
them
in
like
in
a
two
operation
team,
one
for
communities,
but
then
one
for
data.
A
So
why
not
like
you
know,
just
combine
them
together,
so
they
could
be
just
need
to
operate
one
environment
for
all
their
workloads
so
and
also
by
combining
you
know
different
kind
of
hardware
requirements
into
one
you
can
achieve
better
resource
utilization.
You
know
you
can
reduce
cost
all
these
kind
of
benefits.
So
some
background
about
spark.
You
know
just
in
case
you
don't
know
that
so
it's
currently
the
most
popular
framework
for
large-scale
data
processing.
It's
always
often
considered
the
next
generation
Hadoop.
A
Like
Tencent,
like
you
know,
facebook
/
JD,
so
it
has
several
language
bindings
like
you
can
actually
run
spark.
You
can
write
spark
in
Python
in
Scala
in
Java
or
in
our.
It
also
has
a
sequin
interface,
so
you
can
actually
run
sequel.
Queries
on
spark
natively,
so
sometimes
spark
is
called
the
unified
analytics
engine.
A
Well
by
that
it
means
like
you
know,
you
can
actually
run
all
kinds
of
different
batch
workloads
and
streaming
workloads
and
sometimes
machine
weren't,
learning,
workloads
and
processing
workloads,
all
on
spark,
so
you
have
one
shop
for
all
kinds
of
you
know
analytics
needs
so
currently
on
github.
It
has
overnighting
cells
and
starts,
and
over
17,000
Forks,
so
come
together
with
kubernetes.
They
are
basically
two
of
the
most
popular
open
source
project.
You
know
on
github,
so
this
is
a
like
sort
of
like
timeline
of
the
project
history.
A
So
in
November
2016
we
started
the
design
and
soon
after
that,
in
December.
With
these
we
started
that
work
on
in
one
of
English
or
actually
sell
water
reason
why
we
chose
to
do
it
in
a
you
know
fork.
So
you
know
we
wanted
to
be
iterative
iterating
rolling
fast.
So
without
you
know
doing
all
the
code
review
stuff
against
the
main
repo,
so
we
decided
to
do
it
in
a
fork.
A
We
could
start
porting
the
work
we
did
in
a
fork
to
the
man
spark
repo,
so
around
the
timeframe
of
December,
2017
and
January
2018,
so
part
of
it.
Part
of
the
work
that
we
did
in
the
fork
was
upstream
into
spark
2.3.
So
that's
we
that's
one
that
the
work
was
like
you
know,
seen
by
the
mainstream
users.
A
So
basically
people
started
to
talk
about
like
in
the
spark
on
kubernetes,
since
the
release
of
spark
to
point
straight
so
in
October.
Well,
basically,
that
you
know
just
like
about
a
month
ago,
so
we
added
more
featuring
to
spark
2.4
that
all
we'll
talk
about
in
the
later
of
part
of
the
presentation.
A
So
on
a
high
level.
The
native
integration
with
kubernetes
in
apache
spark
basically
gives
you
another
option
to
run
your
spark
jobs.
So
it's
it's
besides
standalone,
yarn
and
missiles.
That
means
actually,
you
can
run
now.
You
have
you
have
the
forest
choices
like
you
know,
so
you
can
actually
run
spark
job
negatively
on
kubernetes,
like
what
do
you
run,
that
on
any
other
coarse-grained
scheduler
backends,
so
the
native
integration
with
kubernetes
in
spark
basically
has
two
parts:
there's
a
community
specific
summation,
client
and
also
there's
a
kubernetes
scheduler
package.
That's
running
the
driver.
A
So
when
it
comes
to
the
submission
client,
it
sector
pretty
straightforward.
So
it
starts
with
a
empty
driver
pass
spec
and
it
applies
a
bunch
of
feature.
Would
we
call
feature
steps
to
the
MTA
drivers,
paw
spec
and
finally,
it
generates
the
the
final
driver
pass
back
and
it
sends
it
to
the
api
server
to
create
the
driver
pod.
So
then
Cooper
need
a
master
or
the
scheduler
basically
schedules
the
driver
power
to
run
on
some
of
the
nodes.
A
So
some
of
the
features
tabs
there
are
things
like
mounting
volumes
or
mounting
secrets
and
also,
like
you
know,
adding
environment
variables
specified
by
the
users.
Those
kind
of
things
so
basically,
each
feature
step
adds
some
part
of
customization
into
the
driver
prospect.
So
the
final
result
is
a
complete
for
driver
prospect
that
you
can
just
send
to
the
API
server
for
Korea
the
pod.
A
So
the
other
part
is
the
scheduler
back-end,
which
is
a
custom
implementation
of
the
coarse-grained
scheduler
back
in
spark.
It
runs
in
the
driver,
so
the
responsibility
of
the
scooter
back-end
is
basically
it.
It
acts
as
a
sort
of
like
a
custom
and
controllers
for
executors
pot.
So
it
manages
these
two
managers,
the
life
cycle
of
these
executor
pots,
so
together
with
the
spark
core
it
does
these
things,
like
you
know,
so
they
handle
task
scheduling,
so
the
sky
dramatic
back
and
manages
the
executor
pods.
So
it
also
wants
you
know
the
executor.
A
A
So
the
tobacco
nature
takes
at
the
spark
configuration
from
the
spark
core
and
also
it
takes
order
to
request
a
new
executor
or
queue
existing
executors
from
the
spark
court,
and
thus
this
thing's
accordingly
by
talking
to
the
APS
over
so
on
a
cluster
site.
So
the
kubernetes
class
are
basically
like
analyze.
A
These
very
basic
services
schedules
and
runs
your
driver
and
Exeter
pods
and
it
provides
networking
support
like,
and
so
you
can
actually
so,
for
example,
if
you
want
to
talk
to
and
remote
HDFS
class
there
or
or
it
also
provides
networking
support
for
the
exit
or
pod
to
connect
to
the
driver
and
yeah.
It
also
provides
like
you
know,
the
login
support,
so
you
can
actually
use
Co
baccardo
logs
to
to
to
fetch
the
logs
of
either
your
driver
or
your
executor
pause
so
under
the
hood.
A
So
when
someone
like
a
user
runs
sparks
on
mate
by
giving
it
a
master
URL
of
the
stars
with
skaters,
so
what
happens
under
under
nice
is
that
so
the
summation
clients
will
run
these
steps
that
just
mentioned
to
create
a
driver,
pass
back
and
send
it
to
the
API
server.
So
once
the
driver
pod
gets
created,
this
the
master
will
schedule
that
to
run
on
some
of
the
nodes
in
a
cluster,
so
once
the
driver
part
starts
running,
will
request
executor
parts
in
mini
batches.
So
what
writers
start?
A
B
A
A
So
accordingly,
so
the
way
or
the
process,
the
executor
pass,
get
created
it's
very
similar,
so
the
driver
politics
and
empty
excerpt
or
pass
spec
and
then
runs
a
bunch
of
steps
on
the
MT
Exeter
pops
back
to
generate
what
are
we
called
a
final
exterior
prospect
and
then
send
it
today
guess
over,
so
this
will
basically
create
on
one
executor
pods.
So
again,
these
steps
do
things
like
you
know.
A
If
you
want
to
mount
some
secrets
to
your
Exeter
power,
so,
for
example,
you
want
to
access
some
remote
services
or,
for
example,
HDFS
security,
defense.
The
secret
might
carry
the
delegation
token
in
that,
so
you
want
to
actually
mounted
that
into
your
your
pods,
so
they
can
actually
use
that
for
act,
remote
access,
so
yeah.
So
then,
once
the
Exeter
part
gets
rescheduled,
it
starts
running
and
we
attach
I
emptied
our
Wallen
to
be
used
for
as
the
working
directory
for
the
executor.
A
So
because
of
the
way
networking
works
in
kubernetes.
So
the
way
the
Exeter
paws
connect
to
the
driver.
Pod
is
also
pretty
interesting.
So
on
yarn
or
on
you
know,
outer
scheduler
pack,
ins
you're,
a
networking
connectivity
is
a
grunted
like
you
know
you
you
would
assume
so
the
Exeter
pods
will
be
able
to
talk
to
the
driver,
pod
natively,
but
kubernetes
things
are
different.
So
actually
the
executor
paths
connect
to
the
driver
pass
through
a
fully
qualified
DNS
name.
A
So
you
know,
like
you
know,
on
communities
in
order
to
have
to
give
apart
a
fully
qualified,
DNS
name,
you
have
to
put
the
part
behind
a
service,
so
this
is
actually
handled
by
the
submission
client.
So
the
summation
client
before
creating
a
driver
pod.
It
actually
creates
a
header,
a
service
for
the
pod.
So
this
basically
gives
the
driver
pod
a
fully
qualify,
DNS
name.
So
then,
so
after
that
you
know
once
the
executors
pass
start,
they
can
actually
use
this
fqdn
name
to
talk
to
the
driver
so
yeah.
A
This
animation
basically
shows
what
happens
like
and
so
the
driver.
So
the
submission
client
basically
sets
up
a
service
for
endpoint
and
driver
for
for
for
the
driver
and
block
manager
in
the
driver
process.
So
this
is
actually
I
had
a
headlight
service
which
was
pretty
straightforward
and
then,
after
that,
the
exterior
will
be
able
to
connect
to
the
driver
using
that
service.
A
So
again,
because
of
the
way
cover
needs
works
like
so,
some
seeing
you
naturally
do
in
outer,
without
our
scheduler
backends,
like
yarn
or
Miso's,
wouldn't
work
in
kubernetes.
One
thing
is
like
no
Yukon
is
because
the
driver
runs
in
the
in
the
cluster
in
a
pod.
It
doesn't
have
access
to
these
application
dependencies
like
jars
or
data
or
config
files
on
your
locker
submission
machine.
So
to
solve
these
problems.
Actually
we
support
these
kind
of
like
to
dependency
management
options.
A
One
is
we,
so
we
basically
provides
a
bit
a
set
of
basic
images
you
can
use
to
build
upon.
So
one
way
is,
you
know
you
can
always
fake
your
dependencies
into
your
custom.
Images
then
use
these
to
run
your
your
as,
like.
You
know,
as
the
SPARC
driver
or
order
executors,
which
basically
already
have
all
the
independencies
locally
visiting
the
container
or
the
other
way
that
we
all
recommend
for
large
amount
of
data.
A
Is
that
you
just
put
your
data
or,
like
you
know,
large
dependencies
like
jars
onto
some
remote
storage
system
like
HDFS
or
Google,
Cloud,
Storage
or
or
or
Amazon
s3.
Then
you,
you,
basically
access
these
remote
storage,
just
like
you
usually
do
so
you
give
them
like
an
HTTP
endpoint
or
a
HDFS
URL.
So
a
spark
itself
knows
how
to
you
know,
access
these
files.
B
A
Remote
application
dependencies
get
handled
by
the
driver
between
spark,
2.3
and
spark
2.4,
so
in
spark
2
points
3.
We
actually
used
an
in
a
container
for
downloading
the
remote
dependencies
in
the
driver.
So
what
happens
like
you
know,
so
the
summation
client
will
ads
and
in
a
container
into
the
driver
pod,
so
these
that
in
a
container
will
be
responsible
for
downloading
these
application
dependencies
like
jars
or
files
onto
a
user,
specify
location.
So
once
you
know
a
successfully
downloads
all
these
things,
it
will
as
all
the
jars
to
the
class
pass.
A
So
you
can
actually
like
it.
Naturally,
you
can
access
the
these
jars
in
your
spark
program,
so
in
spark
2.4
actress
since
the
release
of
spark
2.3,
there
are
some
debates
over
like
net
the
use
of
in
a
container
because
well,
the
spark
community
was
not
very
happy
use
of
this
inner
container,
which
asks
you
know,
complexity.
Also,
some
you
know
more
code
to
min
10.
So
then
we
decided
to
get
rid
of
the
inner
container.
A
So
so
the
solution
will
be
to
actually
run
the
driver
container
so
to
run
the
submits
sparks
ma
to
command.
In
climb
mode
in
a
driver
container,
so
by
doing
that,
we
can
actually
leverage
the
native
support
in
spark
itself
for
downloading
application
dependencies,
because
spark
itself
knows
how
to
download
dependencies
before
it
starts
running
your
program.
So,
but
with
that
change,
there's
a
very
significant
change.
So
first
are
no
longer
it
downloaded
to
a
user-specified
location.
B
A
Rather,
these
files
get
downloaded
into
the
local
working
directory
and
enter
a
randomly
generated
directory.
So
you
can
still
access,
like
you
know
things
like
jars,
natural
light,
because
these
are
these
will
be
added
to
the
class
path.
But
if
you
have
some
file,
you
want
to
access
by
a
file
path,
though,
so
you
have
to
use
the
spark
files
that
get
to
resolve
these
file
paths.
Basically,
so
this
basically
gives
the
list
of
existing
feature
as
of
spark
2.4.
A
So
we
have,
you
know
we
have
the
the
basic
summation,
client
and
the
native
schedule
back-end
in
English
in
a
ripple.
Well
so
also
supports
these
languages,
so
you
can
actually
use
Java,
Scala
Python
or
are
it
also
supports
running
sequel
queries?
If,
for
example,
you
can
actually
set
up
the
sequel
for
gotta
actor?
What's
it
what's,
it
called,
but
there's
some
kind
of
server
you
can
actually
run
in
spark.
Then
you
can.
You
can
send
sequel
queries
to
so
you
can
also
do
that.
So
currently,
I'm
gonna
support
static
resource
allocation.
A
That
means
doctor,
you
have
to
figure
out,
like
you
know
how
many
executors
you
want,
and
so
you
can't
change
the
number
of
executors
at
runtime,
so
it
supports
for
both
container
local
and
remote
dependencies
like
what
it
what
I
just
described.
It
also
supports
these
basic.
You
know
pot
customizations,
for
example.
You
can
add,
like
you
know,
you
can
specify
hard
to
CPU
and
memory
limits
on
the
pot.
So
this
is
actually
one
thing.
A
You
know
the
SPARC
native
in
it
coronated
integration,
it's
different
from
the
yarn
or
missiles
integration
is
that
you
can
actually
use
fractional
values
for
CPUs,
for
both
driver
and
executors.
For
example,
you
can
actually
use
like
a
0.5
or
100
meters
if
use
for
your
driver
or
executors,
which
you
cannot
do
in
outer
security
beckons.
So
you
can
also
say
you
know
these
standard
things
like
label
annotations
or
environment
variables.
For
your
pots,
you
can
also
mount
arbitrary,
kubernetes
secrets
into
into
the
pot.
A
So
it
also
has
limited
support
of
these
basic
walden
types
like
empty
their
host
pass
or
PVCs.
So
you
can
also
use
image
post
secrets
to
pull
images
from
some
private
image
registries
yeah.
So
it
also
comes
to
this
fool,
cluster
and
mobile
support
and
also
limited
climb
of
support
by
that.
I
will
talk
about
that
later.
So
what
I
mean
by
limited,
oh
yeah,
so
in
SPARC
2.4,
we
added
the
climber
support.
This
is
actually
one
of
the
most
requested
features
since
the
release
of
the
SPARC
2.3.
A
So
you
can
actually
run
the
driver
both
outside
or
inside
the
cluster,
but
there's
a
tricky
part
when
you
run
the
driver
also
at
the
cluster,
because
users
are
responsible
for
setting
up
the
network
connected
connectivity
from
your
executor
pod
running
in
the
cluster,
to
the
driver
pod
to
the
driver
running
outside
the
cluster,
so
which
is
not
always
possible.
So
you
have
to
take
that
into
consideration.
A
So
the
recommended
way
is:
if
you
want
to
run
like,
especially
if
you
want
to
run
on
notebook
kind
of
applications,
you
want
to
run
these
things
in
the
class
or
so
the
me
structure.
You
run
your
client
in
the
English
class
or
in
a
pod,
for
example,
we
actually
run
your
attributer
notebook
in
a
pod,
so
then
a
networking
connectivity
part
is
much
easier
to
handle.
So
what
happens
like
you
know,
so
you
can
just
create
a
Heather
service
manually.
A
So
this
is
because
it's
runs
in
a
climb
ode,
so
the
summation
client
won't
be
run
at
all.
So
what
is
the
at
least
a
part
of
that
handles?
The
creation
of
the
driver.
Part
won't
be
run,
so
you
won't,
or
so
the
nabhi
chakra
there
won't
be
an
entity
that
will
create
the
service
for
you,
so
you
have
to
do
that
manually.
So
your
lay
the
process
like
you
create
a
service.
First
use.
You
know
some
label
selector
that
you
will
later
apply
to
the
driver
pod.
So
then
you
start
your
trevor
pod.
A
So
then,
these
the
service,
you
you
create
previously,
will
be
bound
to
the
driver.
Pod,
so
also,
if
you
run
your
driver
pod,
if
you
were
drawing
your
driver
in
a
client
in
a
you
know
in
it
cluster,
you
can
also
leverage
garbage
collection.
So,
for
example,
if
you,
for
whatever
reason
you
delete
your
driver
pod,
so
all
the
exterior
paths
created
by
the
driver
pot
will
be
gone.
They
due
to
a
garbage
collection,
so
oh
yeah,
so
this
is
basically
like
a
no
very
wise
how
it
works.
A
So
you
create
a
featherless
service
before
you
create
your
driver
pot.
So
you
know
the
server
pockets,
a
dns
name
from
the
service,
so
in
Sparks
3.0,
which
will
come
next.
We
also
added
support
for
Kerberos
authentication,
so
you
can
actually
use
that
for
security.
Fsx
access
I
know,
like
you
know,
most
enterprises
who
run
HDFS
on
Burrell
actually
will
have
ostentation
enabled
for
security
purposes,
so
this
is
also
supported
in
SPARC
itself.
A
A
A
B
A
There's
two
ways:
the
first
is
you
just
set:
the
environment
variable
had
to
come
from
there
to
your
local
Hadoop
configuration
directory,
so
the
summation
client
will
basically
check
the
config
in
that
directory
and
generates
a
config
map
and
mount
that
into
your
part.
Well
or
you
can
also
use
your
existing
config
map
that
your
to
have
some,
for
example,
somehow
the
configuration
unit
so
additionally
to
the
east,
like
the
Native
communities,
integration
name,
spark.
We
also
create
work
on
and
creates
this
kubernetes
operator
for
spark.
A
So
this
is
like
you
know
so,
following
the
recent
trend
of
doing
extensions
through
CRT
and
custom
controller.
So
this
is
a
seer,
so
it
Hector
has
230
types,
one
for
one
shot
up
ignition
and
the
other
for
cron
type
applications
and
there's
a
custom
controller
that
basically
runs
sparks
I
made
for
you
and
once
the
application
starts
running
or
start
monitoring
these
applications.
So
it
comes
with
so
one
thing.
One
good
thing
about
this
has
actually
support.
You
know:
Paul
customization,
all
kinds
of
customisations
through
a
mutating
admission
webhook.
So
that
means
dr..
A
So
it
also
does
automatic
setup
for
exporting,
like
your
application
level
and
Driver
exceeder
metrics
to
a
Prometheus
and
also
as
supports
in
college
installation
with
helm
and
well
get
are
seeing
that
comes.
Wizard
is
kamala
to
cost
part
control,
so
you
can
actually
use
that
to
stage
your
local
dependency.
For
example,
if
you
have
some
files
on
your
local
machine
or
jars
on
your
local
machine,
if
you
use
part
control,
it
automatically
is
stages
these
up
dependencies
on
to
a
remote
location.
A
You
specify,
for
example,
you
can
actually
automatically
upload
these
dependency
to
an
s3
bucket.
Then
you
so,
and
the
smart
control
command
like
to
also
automatically
changes
your
dependency
from
your
local
ones
to
the
ones
on
the
remote
location.
So
you
don't
need
to
handle
all
these
things
manually.
So
all
these.
A
For
you
yeah,
so
this
is
one
of
the
example
like
you
know,
so
you
can.
You
can
get
a
sense
on
how
a
CR
our
customer
resource
for
a
spark
application
looks
like
so
you
can
basically
specify
how
the
driver
looks
like
how
director
looks
like
and
also
he
can
enable
monitoring
or
integration
with
prometheus.
A
It
seems
like
that.
Okay,
so
talking
about
road
maps,
this
is
what
you
would
expect
from
the
community.
You
know
moving
forward
that
you
probably
was
seeing
sparks
3.0
and
later
so
one
thing
is
to
solve
the
problem.
So
basic.
All
the
pot
customizations
you
just
saw
are
danced
through
new
spark
config
options,
but
what
the
problem
is
like?
No,
so
because
there
are
so
many
different
things
you
can
customize
with
smart
power
with
community
as
part.
A
Well,
so
if
we
keep
doing
that,
whoa
we'll
end
up
with
so
many
different
new
options,
that's
not
something
we
want
to
do,
because
we
have
to
maintain
these
spark
off
options
moving
forward,
so
we
decided
to
adopt
new
top
a
new
approach.
So
we
basically
allow
you
to
specify
a
plot
template
for
poster
driver
and
exterior
parts.
Now
you
can
do
what
kind
of
a
custom
customization
or
configuration
you
want.
A
So,
for
example,
you
can
actually
specify
pod
affinity,
affinity
or
anti
affinity,
or
you
can,
like
you
know,
use,
tolerant
aberrations
or
or,
like
you
know,
adder
things
like
security
context.
So
then
you
just
give
the
summation
client
these
templates,
so
the
summation
client
will
use
these
template
to
create
the
final
driver
and
exterior
parts
back.
So
the
another
thing
which
is
often
requested
is
the
support
for
dynamic
resource
allocation.
That
means,
actually,
we
support
changing
the
number
of
executors
based
on
the
word
on
the
load
of
the
application
at
runtime.
A
So
this
were
Act
requires
a
new
shuffle
service
that
can
work
natively
with
Sparta
with
communities.
We
actually
did
this
in
our
fork,
but
the
way
it
was
designed
has
some
problems,
so
we
decided
to
not
upstream
that
into
the
main,
smart
repo.
So
we
are
currently
working
on
a
new
shuffle
design
that
well
features
something.
We
call
these
aggregated
storage.
A
That
means
actually
use
separate
storage
out
from
your
compute,
so
we
also
want
to
work
on
better
support
for
local
dependency,
like
what
I
just
mentioned,
like
you
know,
so
we
currently
only
support
two
ways
for
using
dependency.
One
is
you
bake.
Did
that
mean
to
your
custom
images
or
you?
You
know
upload
them
to
whatever
remote
storage
you
want
to
use.
So
another
thing
is
we
want
to
investigate
better
scheduling,
support.
One
of
the
issue.
A
Was
that
the
way
the
kubernetes
schedule
works
is
you
might
have
the
resource
to
just
run
the
driver
pub,
but
you
don't
have
resource
to
run
an
aide
any
of
the
executor
parts.
So,
in
our
case
your
application
basically
haunts
because
the
driver
part
is
not
doing
it
won't
do
anything.
It
was
states
stay
there
on
tutors
resources
to
run.
At
least
you
know
one
Exeter
power
to
run
your
tasks
so
because
I
don't
why
I
see
the
driver
power
just
stay
there
forever.
A
So
this
is
something
want
to
work
on
well,
this
is
particularly
an
issue
for
small
class
or,
like
you
know,
you
have
a
class
or
with
only
three
or
five
notes.
You
have
very
limited
amount
of
resources.
In
that
case,
if
the
driver
is
started,
but
there's
no
resource
for
any
of
the
executors
pause,
then
you
you're
in
a
very
bad
position.
A
So
if
you
want
to
get
involved
nurse
doctor
a
couple
like
a
couple
ways,
so
the
Dakotas
enter
the
directory,
a
resource
manager,
slash
Community
Center,
you
know
the
spark
called
ripple
so
the
documentation
you
can
actually
find
the
kubernetes
specific
documentation
on
the
facial
spark
doc
site.
So
we
also
have
you
know
the
spark
user
and
that
main
enlist
you
can
actually
post
questions
or
feature
requests
to
to.
B
B
Sorry,
just
a
quick
question,
because,
before
what
we
at
least
I
will
be
taught,
is
a
busy
Hadoop
when
they
trying
to
distribute
the
workload
they
were
distribute
to.
They
will
give
you
the
workload
to
the
closest
the
data
where
the
data
resides,
but
so
far
like.
If
you
know
you
have
an
interface
between
that
when
you
assign
the
part
where
you
also
you
know
inherit
that
kind
of
capability,
so.
A
Yeah,
that's
a
very
good
question.
Actually,
so
we
did
this
in
the
fork.
But
again
so
we
saw
the
design
wasn't
really
very
good.
So
when
we
upstream
that
working
to
into
the
sparkman
ripple,
we
decided
to
not
assume
that
part
we
actually
had
support
like
you
know
so.
Data
locality
supporting
the
fork,
but
it's
not
in
the
spark
men
ripple
yet
so
that
still
needs
to
be
worked
on
yeah.
A
Ic
got
it
so
the
question
is
about
like
when
you
want
to
like
know
so.
Placement
of
these
driver
or
executor
pass
on
to
notes,
given
the
limited
amount
of
resources
on
the
node
right.
So
that's
a
wonderful
reason,
like
the
main
reason
why
we
wanted
to
so.
We
actually
saw
in
spark
just
two
points.
Three,
we
didn't
allow
fractional
CPU
adders
for
the
exterior
part,
so
you
have
to
use
a
minimum
lei
once
if
you
for
your
executor
power,
then
we
discover
like
those
so
for
small
clusters.
A
There
might
not
even
be
resources
to
run
your
any
of
the
extra
parts.
So
then
we
actually
added
support
for
using
fractional
values
for
for
CPU
request
and
limits.
So
you
can,
like
you
know,
maybe
beam
packing
like
in
a
more
Exeter
pass
onto
the
same
note.
Oh,
this
is
what
super
help
of
for
small
clusters,
so
my
suggestion
would
be
to
like
you
have
to
do
benchmarking
for
your
application
and
see,
and
you
know,
given
the
limited,
CPU
and
memory
requests
for
the
container,
whether
you
achieve
you
know
good
enough
performance.
B
A
It's
not,
then
you
have
to
figure
out
like
you
know,
so
what
would
be
well,
especially
if
you
have
a
very
limited
amount
of
memory
available,
you
might
run
into
GCC
issues.
If
you
have,
you
know,
production
like
especially
for
production
jobs.
You
might
so
your
job
might
spend
most
of
time
like
in
between
GC.
That's
something
you
don't
want
to
have
a
good
one,
a
wide
so
yeah
so
well,
for
especially
like
what
I
mentioned
for
small
clusters,
you
want
to
benchmark
your
application,
see
you
know.
D
I
have
two
questions.
Why
is
that?
Can
you
explain
a
little
bit
more
about
your
load
map
said
here.
The
last
one
is
the
schedule,
enhancement
in
three
three,
three,
three
right
so
yeah.
Can
you
explain
with
what
which
topic,
which
kind
of
forecast
will
be
the
enhancement
for
scattering
yeah
another
one?
Is
that
I
the
spark
job
may
be,
in
my
opinion,
either
way.
D
A
So
for
your
first
question:
actually
we
we
are,
we
were
talking
to
a
project
called
cool,
be
back
I,
don't
know
if
you
have
heard
of
that
or
not
it's
actually
a
project.
Undercover
daily
seek
scheduling,
so
the
goal
of
that
is
actually
to
support
gun
scheduling.
So,
like
you
know,
so
it's
schedule
a
set
of
pot
or
at
once
were
known.
A
So
we
talked
to
them,
but
it
turns
out,
like
Eric,
didn't
really
work
for
a
for
our
use
cases,
because
so
what
we
really
want
is
there
is
a
current
II
that
you
can
actually
schedule
one
driver
pod
and
at
least
one
Exeter
pot.
So
that's
the
minimum
requirement
we
have.
But
what
is
the
thing
is
like
the
Trevor,
pod
and
extra
pot?
They
don't
run,
they
don't
start
at
the
same
time,
so
you
can't
roll
a
schedule
them
as
a
batch.
A
So
you
have
to
start
the
Trevor
Plouffe
first
and
then
the
driver
pod
will,
you
know,
create
the
exit
so
yeah.
Well,
I
guess
what
was
still
talk
was
done
like
no
see
if
there's
any
chance,
they
can
a
support
for
this
kind
of
I
use
cases.
I.
Don't
think
this
is
unique
to
Sports
Park.
So
for
order,
like
you
know,
some
master/slave
kind
of
work
clothes
there
might
be
so
that
the
same
problem
might
apply
to
these
use
cases
as
well.