►
From YouTube: Spark on Kubernetes: Best Practice and Performance - Junjie Chen & Jerry Shao, Tencent
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Spark on Kubernetes: Best Practice and Performance - Junjie Chen & Jerry Shao, Tencent
As of version 2.3, Spark can run on clusters managed by Kubernetes, the de facto automation framework for contained based applications which is a significant milestone for k8s to support big data services. In this talk, firstly we will introduce our work for offering spark service via Kubernetes deployment as public cloud services, like: Authorization and Logging, and multi-tenancy through namespace and quota management of Kubernetes, etc. Then we will share the best practices of performance tuning details while running Spark application, includes: tuning detailed configurations from Kubernetes and Spark for maximum resource utilization, integrating with zookeeper service to achieve high availability, etc. In prospective of performance, the TPC-DS workload is used to present performance impact brought by configurations change.
To learn more click here: https://sched.co/FuLs
A
And
center
my
name
is
detergent
year
and
today
we
will
be
delivering
his
presentation
about
spark
on
kubernetes
Watson
practice
spark
is
a
very
popular
or
the
most
popular
Big
Data
framework
and
kubernetes
is
the
most
popular
cloud
management
platform.
So
Tencent
wants
to
do
something
to
combine
these
two
together
and
today.
Ginger
and
me
will
share
our
experiences
on
the
kubernetes
and
spark
before
we
begin
I'd
like
to
run
a
small
task,
so
I
using
sparklies
folder.
A
B
B
A
The
most
popular
dictator
framework
so
far,
so
why
is
it
so
popular
because
it
has
some
advantages?
The
first
is
is
based
on
the
daj.
It's
a
teacher
based
on
a
scheduler.
We
know
that
conventionally,
before
Google
released
it
it's
a
mass-produced,
thesis
and
says
the
mass
producing
history
based
on
linearization
and.
B
A
A
master
after
the
map,
however,
in
the
complex
scenarios
there
will
be
problems
with
the
nodes
and
also
redundant
steps.
So
how
to
improve
this
answer
is
Sparkle
turns
the
linearization
into
tree
type.
It's
like
a
data
scheduler
and
it's
a
very
big
advantage
of
what
spark
and
the
second
advantage
is
if
it
has
a
cache
API
or
a
memory
computation
at
the
early
stage
of
spark.
It's
selling
point
it's
it's.
A
It
says
it
sorry,
memory
computation,
well,
memory.
We
know
like
normally
it's
a
GPIO,
OTV
lipo,
so
memory
turns
into
our
disk
or
a
hard
drive.
How
can
we
utilize
memory?
Better?
Spock
is
one
approach
and
it's
a
framework
based
on
memory.
So
that's
the
second
advantage
of
Spock.
The
third
advantage
is
how
to
better
utilize
memory.
Memory
is
a
huge
and
when
GBM
is
runny
memory,
there
are
problems.
There
are
better
solution.
A
A
A
Older
requests
will
go
to
the
drive
node
and
there
will
be
translated
into
the
language
that
spark
an
answer
like
a
job
or
stage
than
the
SQL
and
the
it's
a
distributor
engine.
So
it's
called
as
cuter.
The
acutest
function
is
to
receive
that
has
a
front
driver
and
then
to
ask
you
to
the
job
and
then
return
it
back
to
the
driver.
So
this
is
the
interaction
between
driver
and
execution
in
spark
and
with
our
distributed
architecture,
can
we
use
it
on
our
cluster?
A
Here's
a
key
is
the
yellow
one.
We
introduced
a
concept
of
cluster
manager
and
our
process
needs
to
be
scattered
to
the
cluster.
We
need
to
disappear
resources
to
the
cluster.
That's
why
we
need
these
cluster
manager
to
start
these
application
or
started
these
spots
distributed
to
architecture.
So
SPARC
is
very
simple.
The
architecture
is
very
simple,
and
here
I
want
to
say
spark
at
the
beginning
of
the
design.
It
has
two
layers.
One
is
how
to
schedule
the
spark
itself
and
second,
how
to
schedule
the
task
in
applications
of
spark.
A
A
A
Imposter
meters
is
a
very
popular
dish.
Os
and
it's
the
same
level
with
sparks
so
of
harsh
meters
has
a
threshold
or
it's
a
bit
complicated
as
a
result
to
promote
spark
better
to
reach
more
clients,
but
in
the
1.7
or
1.5
version
it
launched
a
light
way
to
build
in
class
a
measure
and
it's
it's
inside.
The
spark
is
called
a
stand-alone
or
cluster
manager,
and
it's
not
a
modern
cluster
clap
aggregator
it'll,
simplify
to
class
and
as
a
French
functionally
to
schedule
the
process.
A
It
does
not
have
any
function
for
the
function,
isolation
or
the
line
up
all
the
scheduling.
It's
very
simple:
it's
just
a
class
matter
and
it's
lightweight
and
where
the
mature
of
a
head
hub,
we
have
a
head
of
2.00
and
more
clients,
more
companies
that
are
operating
there
had
opportunity
to
point
at
all
or
two
point
four,
so
spark
unknown.
A
A
So,
in
now,
in
a
part
of
ranch
we
can
the
the
spark
cannot
support
to
the
kubernetes.
Having
said
that,
why
do
we
need
kubernetes?
Why
do
we
need
to
support
a
communities?
First
of
all,
communities
has
a
lot
of
advantages.
Well,
computers
is
based
around
our
container,
so
it
has
the
advantage
of
talkers.
First
is
the
portability
conventionally?
A
The
delivery
or
the
development
of
software
has
a
big
problem,
and
the
development
environment
is
very
good,
but
once
it
is
surprised,
then
problems
come
up
and
that
has
traveled
lots
of
developers
and
docker
is
designed
to
solve
these
issues.
To
make
sure
the
implementation
of
the
deployment
will
be
very
smooth
and
the
advantage
of
Dockers
also
can
also
be
seeing
in
kubernetes
and
dokin
kubernetes
house,
also
as
many
other
ages,
which
mean
a
lot
for
the
companies
and
especially
like
the
isolation
of
resources.
A
Lots
of
features
are
very
good,
and
last
but
not
least,
it
has
an
open
source
community,
which
is
very
active
and
I
believe
you
have
used
a
lot
of
open
source
software
and
some
are
small
and
some
are
hard.
Some
are
big
so
how
to
recharge,
whether
it's
a
good
community
or
whether
it's
a
good
open
source
platform,
then
the
most
important
the
criteria
is
we
look
at
the
community
if
the
community
is
good,
then
that
means
the
open
source
platform
is
very
good.
A
A
B
A
Know
that
in
traditional
companies,
when
they
try,
they
will
turn
to
Combinator's
for
micro
surveys,
and
they
were
two
primary
surveys
in
common.
It
is
like
some
small
web
services,
but
there
is
a
problem
if
we
want
to
run
some
big
data,
we
need
another
cluster
or
we
need
a
standalone
cluster
manager.
B
A
C
C
B
C
C
C
Very
similar
to
other
architectures
when
you're
handing
a
spark
application
and
after
that
there
will
be
a
client
interaction
with
it.
The
service
or
part
will
be
started,
and
then
this
process
will
begin.
So
it's
very
simple,
so
we
decompose
it
to
see
the
details
for
the
first
step.
We
need
to
have
such
a
sparked
application
as
the
sparks
a
meet
remotely
when
you
submit
that
you
need
to
submit
to
to.
B
C
B
B
C
B
B
B
C
B
B
C
B
C
The
application
on
humanity's
they
are
run
in
containers,
so
we
need
to
know
what
is
the
image
for
container,
and
here
we
have
the
image
for
container
for
smart.
When
you
click
on
the
button,
it
will
begin
to
run
it's
very
automated.
What's
very
easy
to
use,
you
don't
need
to
know
much
knowledge,
Y
Combinator's,
and
you
can
run
it
smooth
judo.
B
C
C
C
B
B
B
C
B
B
B
C
C
C
B
C
B
B
C
C
C
B
B
B
C
C
C
B
C
C
B
C
B
B
A
B
B
A
Is
the
surface
findings
in
highs?
There
are
two
ways
once
a
we
configure
the
surface:
they
are
to
suit
scenarios.
One
is
a
weekend
visit
through
the
environmental
configuration
and
second
is
through
the
HTTP,
and
this
is
our
configuration
on
sparklin
and
next
we
look
at
the
sparklings
and
copper
natives
and
whether
it's
different
around
its
performance,
Anyang
I.
B
A
Introduced
we
have
two
data
and
processing
models,
one
is
separated,
storage
and
the
processing-
and
second
is
the
storage-
is
put
together
with
the
kubernetes.
So
here
we
look,
we
used
two
environments
and
we
can
tell
them
the
first.
The
environments
is
HDFS
inside
the
kubernetes
and
in
this
environment
we
can
see
there
are
lots
of
HDD
and
seconds
environment.
A
A
A
B
A
Spark
of
that
sicko
the
partition.
This
is
to
control
the
sparks
of
parallelism.
We
started
out
of
1.24,
so
it's
a
related
to
the
memory.
A
secure
man
memory
and
his
based
on
experience
and
on
downside
is
how
to
pass
beneath,
and
this
is
to
join
the
small
table
with
the
big
table.
If
the
small
table
is
less
than
30mb,
then
it
will
be
broadcasted
to
the
big
tables.
A
B
B
A
B
B
B
B
B
A
B
A
Piece
of
data
so
first
offer
is
work
visit
the
memory
to
see
what
well
the
block
is,
and
after
you
to
get
an
IP
e
like
the
data,
a
data
noted
one,
so
it
will
execute
or
distribute
a
task
a
closing
today,
so
it
then
it
gets
the
benefits
of
rotational
13.
Second,
is
the
container
data
locality
in
Jung.
B
A
A
B
A
We
have
done
lots
of
work
and
we
have
shown
you
we
first
sat
what
is
a
spark
and
the
kubernetes
and
a
fir
of
communities
and
our
use
cases
and
optimization
at
what
once
what
I
once
Jazz's
three
points-
and
it's
also
based
on
my
experiments-
is
first
in
using
Canadians
as
lots
of
advantages.
First,
is
we
have
a
unified
deployment
environment?
A
This
is
the
other
big
advantage
and
the
second
is
in
our
tests
and
all
our
user
cases
will
fund.
When
we
can't
analyze
the
big
data.
We
don't
see
that
a
waste
of
old
performance
or
a
reduction
of
performance,
but
they
are
like
locality
that
there's
a
problem
but
in
the
containerization
kubernetes
is
very
hard
to
realize,
at
least
for
the
moment.