►
From YouTube: A Hybrid Container Cloud With Kubernetes and Hadoop YARN - Jian He & Bushuang Gao, Alibaba
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
A Hybrid Container Cloud With Kubernetes and Hadoop YARN - Jian He & Bushuang Gao, Alibaba
Hadoop YARN is a resource management platform to run big data applications such as MapReduce, Spark and it is architecturally different from Kubernetes which well suits long running services. Many organizations keep both of them to fulfill different types of workloads. However, this approach will incur staggering ops and hardware cost. Looking at the differences in the 2 types of workloads, is it possible to let them share a single cluster while keeping both resource management systems working in harmony? What are the requirements and what hurdles do we need to overcome? In this talk we will present a framework developed by Alibaba, that can seamlessly run both Kubernetes and Hadoop in a single cluster with the ability of elastic resource sharing. In addition, we will also share lessons we learned in managing both workloads in production to support Alibaba massive commercial platform.
To learn more click here: https://sched.co/FuL1
A
Hello:
everyone,
my
topic
is
about
kubernetes
yarn
and
the
combination
of
kubernetes
and
the
gentleman
first.
Let
me
introduce
myself
actually
I'm
from
Alibaba
Sigma
Sigma
teams
in
charge
of
the
online
dispatch
system.
So
now
I
mean
doing
the
big
data
companies
and
we're
trying
to
develop
yarn
in
our
companies
and
I
used
to
work
in
Hadoop
community
for
several
years
in
charge
of
the
community
maintenance.
So
today,
I
want
to
talk
about
the
yarns
about
the
wall
and
why
of
colocation?
A
And
why
are
they
able
to
do
this
and
then
I
want
to
talk
about
an
architecture
based
on
kubernetes
and
then
based
on
my
past
experience,
I
would
like
to
talk
about
the
kubernetes
of
yarn,
this
hybrid
architecture,
online,
offline
dispatch,
advantages
and
disadvantages.
Lastly,
I
will
summarize
the
these
two
different
platform.
A
First
of
all,
why
we
choose
colocation
in
Alibaba.
Actually,
we've
done
some
service
and,
according
to
the
statistics,
the
data
center,
a
utilization
of
Alibaba
before
a
colocation,
is
very
low.
The
CPU
utilization
was
only
10%
before
we
adopt
colocation,
and
that
is
a
very
low
utilization
rate.
We
waste
a
lot
of
capacities
according
to
Gartner,
they
have
talked
about
the
80%
role
of
Gartner,
which
means
that
80%
there's
only
ITU
sort
of
the
bastion
are
those
I
see
on
top.
A
So,
just
like
the
high
expressway
here,
you
can
see
in
the
picture
on
the
left
hand,
side
on
the
one
directions
you
can
see:
there's
a
car
traffic
channel
the
other
side
on
the
other
side,
actually
there's
not
so
much
traffic,
so
which
means
that
we
need
to
fully
leverage
the
capacities,
our
workloads
in
the
Alibaba
data
centers
through
it
split
into
two
different
topics.
First,
we
have
online
service,
including
the
following
agenda.
A
First,
the
online
service
in
charge
of
the
commercial
platforms,
which
is
the
web
application
that
we
use
daily
in
tall
baltimore,
retail
apps
for
man
and
other
other
search
capacities,
and
also
the
online
servers,
are
charging
in
charge
of
take
on
the
payment
platform,
for
example
on
the
play
and
then
in
the
online
service.
You
also
have
the
middle,
where
the
cue
servers,
the
cache
servers
and
other
database.
A
So
in
Alibaba
we
believe
that
we
have
already
100%
financed,
which
means
that
all
the
things
were
using
pot
container
technologies
on
the
Sigma
hot
management
and
also
there
are
a
lot
other
workloads
which
is
offline
dispatch,
for
example,
jobs.
When
it
comes
to
jobs,
we
have
Big
Data
patch
jobs,
batch
jobs,
for
example
mr
and
otps,
which
is
Alibaba
own
projects.
Own
tools-
and
you
have
think
blink
is
about
real
time.
Calculation,
real,
real-time
computing.
A
We
have
some
legacy
issues,
for
example
the
Sigma.
This
is
our
team.
We
were
in
charge
of
the
online
platforms.
Also
have
fousey
Fuji
are
in
charge,
is
in
charge
of
the
offline
jobs,
a
Sigma
in
charge
of
Tom,
a
team
out
our
search
and
middleware,
which
is
online
jobs
and
Suzy
in
charge
of
offline,
which
is
OD
PS
and
our
sparks
and
jobs.
So
Sigma
is
the
same
kubernetes
with
super
kubernetes
enthusiasts.
The
same
with
yarn
as
yarn
is
also
in
charge
of
the
offline
jobs.
A
So
there
this
is
a
compare
and
contrast
about
online
service
and
offline
jobs,
so
I
have
several
different
features,
as
you
can
see
on
the
slides
online
service
when
it
comes
to
online
shopping,
Web,
Apps
and
the
payment
service,
the
advantages
it's
very
sensitive
to
latency,
for
example.
If
you
want
to
buy
something,
you
need
to
finish
the
into
end
transaction
within
several
hundred
milliseconds.
However,
when
it
comes
to
offline
batch
jobs,
it
is
very
sensitive
to
the
latencies
for
some
offline
jobs.
It
takes
one
or
two
hours
to
finish
the
transaction.
A
It
is
very
also
costly
and
also
the
online
service
is
very
high
priority
for
us.
So
if
there
are
some
accidents,
we
need
to
give
priority
to
the
online
jobs,
so
it
is
in
the
forefront
of
dealing
with
customers.
However,
when
it
comes
to
batch
jobs
offline,
it
is
very
low
priority
because
we
don't
contact
clients
in
the
best
jobs
and
also
the
traffic
pattern
difference
between
online
and
offline
when
it
comes
to
online
service.
A
A
There
are
some
application
applications
or
other
offline
jobs,
so
we're
doing
it
at
night.
And,
lastly,
when
it
comes
to
fault
tolerance,
these
two
are
quite
different.
When
it
comes
to
online
service,
we
need
high
availability
high
resilience
when
there's
a
failures.
It
is
zero
tolerance,
for
example.
If
they're
a
client
you
aren't
buying
online,
you
don't
want
the
system
to
fail.
However,
when
it
comes
to
offline
bad
jobs,
if
the
task
failed,
then
we
need
to
do
a
retry.
It
is
acceptable.
It
has
a
very
high
low
or
high
fault
tolerance.
A
We
need
to
do
retry
and
it
is
acceptable
when
there's
a
sale
and
retire
so
because
of
on
and
offline
have
these
different
complement,
beautiful,
complementary
relations.
So
then,
if
we
can
fully
leverage
these
features,
we
if
we
can
run
it
on
the
same
platform
based
on
the
same
infrastructure,
then
we
can
fully
leverage
all
the
resources
without
any
waste.
A
So
over
the
past
two
or
three
years
within
Alibaba,
our
Sigma
team,
trying
to
have
a
colocation
clusters,
come
up
with
these
colocation
clusters
to
have
this
hardware
that
can
be
adopted
to
Sigma
and
the
SUSE,
and
we
can
do
the
same
maintenance
based
on
the
same
infrastructure
in
the
past,
but
also
published
a
paper
talking
about
colocation.
He
said
it
is
a
production
and
non
production
jobs
combined
together.
A
If
we
combine
production
and
non
production
jobs
together,
then
we
need
to
invest
20
or
30
more
additional
so
because
production,
jobs,
production
tasks,
it's
the
same
with
our
online
service,
there's
a
spike
and
they
need
to
reserve
all
the
resource
for
those.
In
spite,
but
most
of
the
time
the
resource
was
wasted
was
not
fully
leveraged,
so
non
production
jobs
can
actually
use
those
additional
resource
to
run
their
jobs.
In
this
way,
we
can
fully
leverage
the
utilize
they
improve.
The
utilization
rate.
A
A
So
when
it
comes
to
colocation,
we
have
actually
make
some
comparisons
in
terms
of
CPU
utilization
in
before
colocation,
the
CPU
utilization
rate
was
only
around
40%
after
the
colocation
was
adopted.
We
improve
to
40
percent,
so
in
the
past
was
only
is
10%,
so
we
improved
by
30%.
So
we
think
not
about.
This
is
a
huge
improvement.
There
is
a
qualitative
improvement
with
colocation
higher
and
also
we've
done
some
statistics.
The
RT
impact
on
online
service,
the
RT
impact,
can
be
maintained
within
5%.
A
Previously
I
was
talking
about
the
Alibaba
status
quo,
so
colocation
technologies
in
Alibaba
I
was
there
was
fairly
mature
compared
to
them.
We
have
actually
whispered
to
the
test
of
different
rounds
of
w11
events,
so
during
w
levin
our
colocation
technologies,
with
due
to
merely
tons
of
thousands
of
transactions
and
data.
So
if
we
can
actually
make
it
open
source
and
feed
it
back
to
the
community
or
make
it
provided
to
other
same
scale,
operators
who
care
about
cpu
utilization,
then
it
will
be
very
useful.
So
we
have
two
important
project
versus
kubernetes.
A
B
B
B
It
is,
division
are
different,
so
what
I'm
talking
about
main
being
different
from
what
you
have
seen
the
front?
Deleverage,
it's
a
kubernetes
standard,
kubernetes
and
a
variety
standard.
Yon
the
cluster
and
in
the
middle
is
close
to
master
and
agent,
and
also
the
consoles
that
we
developed
targeting
the
hybrid
at.
A
B
Beginning
for
the
whole
cluster,
during
the
initiation
by
a
council,
we
deliver
commands
to
colocation
master
and
to
agent
for
each
cluster
a
machine.
We
will
have
a
ratio
like
40%
to
kubernetes
and
62
yon.
It
was
kind
of
a
ratio
and
after
it
is
completed,
each
of
the
cluster
were
all
developed
and
according
to
and
some
of
the
fires
will
follow
the
traditional
a
pipeline
and
some
go
to
the
new
and,
of
course
after
they
are
all
about
to
the
coefficient
of
hasta.
Well,
dynamically
equal
makes
up
a
vehicle
like.
B
A
A
B
B
B
A
A
A
B
B
B
B
A
B
A
B
Need
to
buy
lots
of
machine
to
meet
the
demands
of
the
data
traffic
during
the
promotion
day
that,
after
that,
they
will
very
likely
not
be
using,
which
is
waste
of
resources.
But
now,
during
this
hybrid
we
can
achieve
auto-scaling
old,
also
bring
online
services
services
and
the
green
and
the
orange
are
offline
jobs.
Before
the
promotion,
we
will
manually
adjust
this
green
parts
to
a
digest.
That
is
how
we
did
actually
during
the
double
level
which
just
passed.
B
Past
the
value,
if
it's
above
the,
if
it's
below
the
parameter,
then
we
will
also
adjust
the
tasks
gradually,
which
is
also
related
to
dispatching
and
scheduling.
Because
when
you
are
making
other
adjustments,
you
must
make
sure
that
the
existing
services
must
make
meets
the
demands,
but
rather
than
that,
you
cannot
do
all
the
services
to
a
single
machine.
B
And
last
I
would
like
to
talk
about
because
personally,
I
have
for
the
development
experience
in
the
two
projects.
So
I
would
like
to
tell
the
differences
from
there
to
Diane
principle.
First
is
kubernetes.
First,
it's
a
chart
of
the
dispatching
I
think
it
is
more
related
to
operation
and
maintenance.
We
offer
a
final
stage
and
assistant
helps
to
adjust
the
current
stage
to
the
final
state,
but
for
yacht
is
designed
for
the
best
job.
The
batch
job
has
a
first
cause
a
vacation
concept,
so.
A
B
B
B
B
B
A
Centering
applications
when
we
are
doing
to
dispatch
you
choose
queue
and
then
choose
the
most
prioritized
queue,
and
also
it
has
a
queue
half
capacities.
You
know,
so
it's
very
complex
for
logic,
so
you
would
choose
the
best
pure
and
then,
after
the
queue
is
true
and
then
he
will
use
the
user
and
then,
after
the
stage
this
is
describing
the
scheduling
sequence
of
yarn,
first
queue
attend
user
and
then
application
so
there's
priority
for
applications.
A
Kubernetes
don't
have
application
priority,
but
yarn
have
application
of
priorities
after
we
choose
applications
and
then
choose
a
container
request
or
similar
to
pot
so
for
Power
has
also
priorities.
So
it's
like
a
top-down
seller
schedule
in
sequence.
The
reason
is
that
we
were
doing
it
as
dispatch.
A
A
So
the
communication
kubernetes
is
quite
different.
Listeners
cuban
artists
is
based
on
api
server,
wash
mechanisms.
All
of
this
was
in
the
everything
was
stored
in
the
edca
etcd.
If
the
etcd
is
like
the
brain
health
communities,
but
for
yarn
when
it
comes
to
communication,
yarn
is
based
on
RP
RPC,
there's
only
application
that
will
metadata
persisted.
So
the
benefit
of
this
is
that
one
cares
doing
the
recover.
A
You
don't
have
to
put
all
the
things
into
a
persistent
data
storage
and
take
this
will
lower
the
latency
and
after
the
recovered,
actually
is
from
a
memory
state
into
a
piers
and
there
you
can
get
information
from
peers
and
then
reconstruct
to
their
schedule.
So,
however,
kubernetes
put
it
all
of
that
into
etcd,
it
is
also
very
easy.
You
can
get
information
that
you
want
from
eg
CD.
So
this
is
the
difference
between
Cuban
ideas
and
run
with
yarn
is
a
big
collection,
so
collectible
performance
is
not
so
good.
A
Compare
those
two
bananas
next
is
the
container
runtown
cuba
dad
is
container
round
tongue,
is
a
CRI
compatible
and
it
is
dr.,
is
using
dr.
technologies.
However,
yarn
have
darker
and
tarball
so
probably
you're,
noting
that
there
was
tar
ball,
but
when
you're
just
doing
or
calling
up
jobs,
it
is
not
containerized,
which
means
that
on
each
knot
you
need
to
put
a
code.
Artifact
and
computations
files
need
to
localize
into
a
local,
not
so
for
yarn.
It
is
using
a
resource,
localization
logic.
A
It
is
very
complex
and
it
is
related
to
different
customers
access
to
if
the
customers
right
private
control,
they
want
to
private
control
to
their
own
resource
and
each
users
can
only
access
to
your
own
resource.
So
that
is
their
requirements.
However,
when
it
comes
to
kubernetes
using
docker.
So
all
of
these
was
uploaded
to
dr-
and
this
is
a
not
product
so
previously
have
talked
about
kubernetes
and
darker
difference.
A
I
want
to
do
a
summary
of
our
session
here
so
because
colocation
in
Alibaba
is
fairly
mature,
but
it
was
not
fully
used
in
all
the
capacity
of
the
Alibaba
and
the
Fuji,
and
the
Sigma
team,
historically
speaking,
were
developed
by
two
different
teams.
So
if
we
want
to
do
in
Alibaba
cool,
it
still
takes
a
lot
of
time.
A
Then,
of
course
our
collocation
technologies
is
a
contribution
by
Fuji
and
the
Sigma
teams.
So
we
hope
that
we
can
really
make
it
open
file
open
source
to
make
all
the
big
manufacturers
just
like
Alibaba,
can
adopt
this
technologies
lower
their
resource
waste,
and
that
is
the
end
of
my
presentation.
Thank
you.
A
Again,
generally,
we're
too
large
to
mutants
are
I'm.
Sorry,
the
audience
is
not
using
the
microphone.
The
interpreter
cannot
hear
the
accent.
Question.
Ok,
so
well,
the
question
is
about
whether
we
use
yarn
over
K
or
Burnett
Asst.
Well,
actually,
it
is,
let's
think
of
this
way.
Young
and
the
true
Burnett.
Is
there
changing
the
dispatch
method?
You
thing
on
to
complement
kubernetes
party
yarn
didn't
solve
the
resource
allocation
issues.
If
you're
using
yarn
over
kubernetes
I
believe
collocation
can
achieve,
does
can
help
you
to
alter
dispatch.
A
Your
users,
so
yarn
over
K
Burnett,
is,
can
help
you
to
dispatch
and
in
the
industry,
yarn
been
in
the
industry
for
many
years.
A
lot
of
companies
have
a
whole
set
of
develop
systems
based
on
yarn.
So
if
we
really
want
to
force
to
use
the
yarn
over
to
Burnett,
is
it
takes
ex-police
to
finish
the
job,
but
I
believe
that
these
two
are
not
contradicted
with
each
other.
A
A
A
All
right,
so
the
question
is
about
offline
and
online.
Well,
the
impact
networks.
Yes,
I,
think
that
our
teams
have
already
have
online
or
offline
control
package.
So,
internally
speaking
in
the
Alibaba
groups,
we
actually
have
standards
for
online
offline
and
there's
a
marks
for
the
only
offline.
So
we
put
it
to
the
portal
and
they
will
recognize
the
marks.
They
know
the
traffic's
and
they
will
auto-tuning
the
online
offline
traffic.
A
A
A
A
A
A
What
I'll
show
my
so?
The
question
is
about
kubernetes
request
model
and
young
request
model
on
a
physical
level.
Are
they
the
same?
So
when
it
comes
to
kubernetes
resource
model,
request
model,
they
have
two
different
classifications
and
the
when
it
comes
to
yarn.
According
to
my
knowledge,
when
I
was
actually
involved,
as
a
programs
previously
there's
a
picture
giving
you
the
CPU
utilization
right,
so
the
fixed
C
apart
from
Sevilla,
the
other
scipio
was
shared
and
capacity
was
shared
by
kubernetes
and
yarn.
A
So
when
it
comes
to
clothes
changing
well
previously,
if
we
change
the
now
sides,
and
then
we
can
help
the
yarn
to
identify
what
kind
of
tasks
need
preemption,
what
kind
of
tasks
don't
need?
So
if
we
need
the
needed
additional
capacity,
it
is
very
hard
because
the
the
external
suit
is
certain
third
parties,
don't
know
anything
about
the
yarns
and
the
kubernetes.
So
we're
trying
to
change
the
size
of
the
node
to
identify
the
task,
and
then
we
then
could
finish
the
task.
I,
don't
know
whether
I
answer
your
question.