►
From YouTube: HDFS CSI Plugin: Speed Up Kubernetes in On-Premises Big Data Cluster - Yi Chen & Junping Du, Tencent
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
HDFS CSI Plugin: Speed Up Kubernetes in On-Premises Big Data Cluster - Yi Chen & Junping Du, Tencent
Kubernetes not only becomes predominant in public cloud area these days, but also becomes a new trend in on-premises big data cluster environment, as an alternative of Hadoop YARN, a resource schedule component. In on-premise big data cluster, majority data are saved in HDFS. How to consume big data in HDFS with Kubernetes is a new challenge to users. In the talk we will introduce our CSI compatible HDFS plugin design and architecture first. Then, we will share our best practices and knowledge about how big data workload Spark use HDFS CSI plugin to access HDFS data when running on K8s. In the end, the TPC-DS benchmark suite will be used to analysis performance comparison between Spark on K8s with HDFS and Spark on YARN with HDFS.
https://sched.co/Nrq7
B
B
A
A
B
A
B
B
A
A
lot
of
the
companies
that
is
related
to
this
fear
also
established
in
2012,
14
and
17,
has
pushed
the
development
affected
into
the
1.0
period.
2.0
in
the
1.0.
The
foundation
has
been
more
mature
and
2.0.
It
has
scheduling
and
also
the
fracturing
of
the
abstract
resources.
After
it
has
been
splitted,
it
can
be
used
on
more
complex,
Big,
Data
calculate
calculation
and
computing.
A
A
B
B
A
History
of
big
data
involvement
from
the
perspective
of
the
fixator
stack,
it
has
been
separated
into
different
levels,
and
we
ago
we
that's
how
we
have
get
mature
big
data
first
NFS,
if
DFS
has
become
the
foundation
of
all
the
vectors.
Of
course,
we
utilize
the
others
like
s3
and
cloud,
and
also
on
trim,
maybe
yourself
that
is
supporting
the
blocks
and
files,
documentation
and
different
type
of
storage
classes,
and
also
the
big
data
workload.
B
A
B
B
B
A
Trying
to
achieve
the
consistency
of
all
of
these
systems,
and
then
is
the
computing
system
in
computing
layer,
including
the
processing
of
the
data
and
computing,
and
also
the
high
energy
like
the
sequel
and
entry
and
spark
as
well.
Just
we
also
mentioned
it's
very
popular
and
also
the
mainstream
technology,
and
it's
utilising
Intel
computing
technology
to
support
many
different
type
of
computing
scenarios,
like
machine
learning
and
also
some
independence
in
areas.
A
B
A
B
A
A
B
B
A
A
A
A
B
B
A
I
think
the
enhancement
of
these
functions
is
that
I
know.
How
can
I
scheduling
my
computing
and
also
so
that
my
computing
happened
close
to
my
data
and
now
I
separated
computing
and
storage
computing
could
be
elastic,
but
storage
is
very
different,
difficult
to
be
elastic.
So
if
we
combine
these
two
part
together,
it
affects
our
business.
How
can
we
realize
this
elastic
computing
and
separate
this
computing
from
the
storage
and
also
prepare
for
the
replicas
I
think
there
will
be
a
very
big
room
for
us
to.
A
B
A
A
B
B
A
B
B
A
B
B
B
B
B
Finish
it
once
and
for
all
it's
a
round
by
round
distribution
model.
Second,
the
big
data.
Sometimes
it's
using
the
IO
fee,
it's
different
from
the
traditional
micro
services,
different
manners,
so
the
different
types
of
workloads
that
mix
them
together
to
distribute
the
good
thing
is
like
a
more
efficiently
integrate
resources,
the
Compu
you
can't
segregate
them
properly
and
SLA
management
to
achieve
the
desirable
level.
B
That
is
a
challenge,
an
important
challenge
for
distribution
and
management
and
thirdly,
honey
and
for
some
scenario,
it's
chewy,
but
for
some
friend
environment
is
not
choose,
isn't
containers,
it's
you
stir
than
desire
and
sound
a
story,
but
they
are
temporary
things.
I,
don't
care
too
much,
whether
they
are
already
or
not
so
overall,
in
terms
of
this,
as
sustainability
is
different
from
the
traditional
SD
WH
or
other.
B
B
B
B
The
purpose
is
to
bridge
the
container
orchestration
between
different
suppliers,
from
different
storage
providers
and
for
providers.
As
long
as
they
follow
this
spec
to
make
the
CSI
driver
of
their
own,
then
it
can
be
utilized
by
other
providers.
It's
not
unique
to
one
platform
and
the
container
orchestration
provider,
as
he
only
needs
to
provide
this
interfaces,
our
interface.
So
all
the
CSI
storage
can
be
used
in
this
cluster.
B
B
It
pays
an
attention
to
the
control,
the
control
flow
of
the
container
orchestration,
using
the
volume
managing
the
volume
and
for
iof.
The
spec
does
not
include
on
iOS.
The
to
designing
criteria
is
to
keep
it
interface
as
central
as
possible,
and
second,
the
distribution,
the
using
of
the
interface.
B
Saya
saya
dividers
service
into
three
groups:
identity
service
at
the
new
services
used
to
expose
the
generic
information
for
the
driver,
like
the
name,
the
version
of
the
driver
and
the
capability
of
the
driver.
So
this
is
a
a
must
service
and
the
second
sub
is
a
controller
service
and
main
function
of
the
first
services
to
cater
to
the
users,
demand
to
create
men
and
manage
our
volumes,
and
thus
a
service
is
not
service
for
some
of
other
disks
and
cloud
disks.
B
This
function
enables
the
attachment
or
the
attachment
detachment
to
with
a
VN.
So
this
type
of
a
service
is.
We
only
need
one
instance
and
that
will
be
sufficient
and
regarding
no
service,
the
main
function
is
to
you
know
the
the
volumes
created
by
controller
service
based
on
the
user's
needs
to
mount
it
on
the
designated
node.
So
it's
a
not
based
service
in
Cuba
Nettie's
and
every
not.
We
require
an
instance
on
every
Cuban
artists,
not.
B
Many
advanced
functions,
one
is
the
brought
volume.
Pattern
is
based
on
flyers,
FS
volume,
and
now
it
supports
the
creation
of
snapshot.
So
you
can
create
a
snapshot
for
a
certain
volume
and
at
a
certain
time,
slot
based
on
his
snapshot,
you
can
recover
or
restore
the
previous
state
and
thirdly,
its
topology
support.
B
Say
size
supported:
this
storage
is
basically,
they
are
mounted
on
the
network
based
volume
and
it's
also
distributed
topology.
So
it's
sensitive
to
the
network
under
the
same
network.
Upon
creation,
this
volume
are
only
used
suitable
for
your
company
couse.
You
know
the
nose
under
the
same
network
and
the
notes
in
some
other
networks
might
not
be
friendly
to
your
applications
or
not
compatible.
So
it's
still
optimized
the
topology
support
before
the
volume
does
not
to
support
a
size,
so
the
volume
is
immutable,
and
now
it
supports
the
resize
of
volume.
B
B
With
your
system,
if
so
volumes
still
stands,
when
the
projects
finished,
you
need
to
manually
delete
it.
So
it
requires
a
manual
deletion
and
this
type
of
volume
the
lifespan
will
be
assigned
with
your
pod.
When
your
part
is
released,
then
the
part
will
also
release
the
volume
as
well
and
the
cycle
of
the
volume.
B
Just
now
I
said,
the
lifespan
of
the
volume
involved,
controller
service
and
non
service
and
for
controller
service
it
is
responsible,
creating
and
deleting
the
volume
why
not
service
is
will
ensure
that
they
are
attached
to
the
corresponding
node,
even
multiple
notes,
while
creation,
it
only
requires
a
a
very
simple
operation.
So
this
is
a
CSI
two
ways
to
deploy.
Csi
services
by
before
I
said:
you're
no
service
on
each.
Not
it's
a
must
have,
and
the
difference
between
the
two
methods
is
whether
you
need
one
or
multiple
volumes.
B
One
method
is
only
need
to
deploy
one
sense,
Phi
the
above
functions
and
second
one.
If
you
combine
the
three
services
and
deploy
it
on
each,
not
it's
also
acceptable
that
within
the
cluster,
you
only
need
one
active
controller
service.
If
you
will
deploy
so
many
controller
services
on
each
node,
you
need
a
strategy
to
you
know
to
maintain
only
one
active,
a
container
at
one
given
time.
Just
like
the
leader
of
HDS,
you
need
you
need
to
maintain
one
active
service
and
this
deployment
would
be
more
intricate.
B
So
we
would
recommend
the
abhava
deployment
approach
so
that
your
your
you
can
maintain
a
simple
architecture.
Next,
let's
review
in
Cuba,
net
storage,
suppose
the
history
earliest
order,
Cuba
netted
supporters,
supported
storage.
They
are
all
maintained
and
developed
by
Cuban
Esther
code
or
all
owned
by
them,
that
with
the
population
of
Cuba
nets,
it
needs
to
talk
more
and
more
providers
vendors.
So
it
came
to
the
Epiphany
that
the
the
model
is
not
sustainable,
so
they
change.
They
shifted
from
the
previous
model
to
the
out
of
tree
auditory
model
to
dark
other
vendors.
B
One
way
is
called
a
flexor
volume.
Another
is
called
si
si
so
far,
Cuba
Nets
no
longer
support.
That
means,
if
you
are
just
freshly
doctor
to
two
minutes,
it
will
no
longer
recommend
you
to
using
Cuba
legs.
It
will
instead
free
recommend
si
si
so
for
some
of
the
intrigue
rankings
give
Annette
is
doing
some
work,
selling
some
to
Otto
tree
and
si
si
model.
So
this
is
a
data
future.
B
Saya
saya
tribe,
a
speck,
is
for
the
storage
of
vendors
and
storage.
Vendors
will
follow
the
specs
to
deliver
to
make
their
own
systems.
So
how
to
require
is
the
suspect
how
to
utilize
this
pack
actually
is
still
a
gap
in
between
this
is
the
drive.
Is
a
deployment
is
not
a
static
process,
it's
a
dynamic
one.
So
that
means
how
can
cuba
nets
dynamically?
B
Identify
that
in
this
cluster,
currently
the
hemolysis
I
Drive,
because
it's
continuously
changing
and
how
to
identify
how
many
existing
dry
storage
drivers
that
are
accessible
and
how
to
develop
the
interfaces
so
Cuba
Nets
provide
in
the
middle
yeah
the
green
ones?
It
provides
many
auxiliary
containers
or
components,
but
taking
a
form
of
containers,
and
these
components
are
functions.
B
Assisting
kubernetes
to
identify
the
thinnest
cluster
that
some
newly
registered
units
and
Ambika
and
using
a
cabinet
is
called
to
distribute
to
the
interface
of
the
driver
of
CSI
driver.
Actually,
the
interface
of
CSI
driver
will
not
be
directly
used
by
the
code,
but
through
an
intermediate,
an
ancillary
component
as
agents.
B
So
through
these
external
components,
these
are
the
lyric
components,
cabinets
can
dynamically
identified
the
CPI
drivers
and
leverage
them
dynamically,
creating
new
drivers
and
corresponding
to
the
to
the
volumes
o
in
a
dynamic
manner.
Just
like
what
house,
cuba
nests
are
called
other
functions,
its
amic
no
need
for
static
configurations.
So
far,
they're
over
60
says
I
try
versus
supported
so
in
Cuban
artists,
and
you
can
visit
our
booth
to
find
out
more
for
CSI
supported
history.
B
A
It's
described
in
the
foot
when
you
describe
the
points
you
need
to
also
define
the
TV
FCC
they're,
sharing
this
image
when
the
pods
is
scheduled
on
to
load
and
it's
going
to
pretty
generous
a
PVC.
If
you
need
to
store
it's
going
to
find
the
corresponding
system
volume
consistent
volume
to
satisfy
your
needs,
for
example,
I
need
100
Gig.
A
A
A
A
A
B
A
B
A
A
A
B
A
Demand
is
that
HDFS
want
to
become
an
in-house
clusters
and
provide
the
unified
solutions
void.
Then
demo
depends
on
the
HDFS
architecture.
If
you
have
listened
to
others
pictures.
Maybe
you
have
been
quite
familiar
with
this
already
because
HDFS
has
been
has
three
replicas,
but
he
fought
for
the
it's
data
storage.
So
it's
very
liable
and
HDFS
has
about
more
than
ten
years
of
experience
and
history,
so
it
has
the
history
of
market
deployment
and
operations
and
utilization.
It
has
been
a
stable
storage
approach
in
it's
very
good
candidates
for
kubernetes
storage.
B
B
B
A
And
their
developer,
actually
the
developer
of
Haru,
her
dude
hdfs,
so
they
have
a
lot
of
experience
of
different
scenarios
in
operations,
expression
and
maintenance
experience
as
well.
Ozone
is
an
object,
storage
system,
so
its
goal
is
to
solve
the
scaling
problem
of
HDFS
so
that
you
can
achieve
a
cloud
native
storage.
B
A
If
we
want
to
put
a
HDFS
and
also
as
its
persistent
volume,
is
there
any
method,
any
path
first
is
utilizing
the
HDFS
and
FS
gateway.
The
NFS
gateway
is
supporting
that
you
are
not
belonging
to
the
HDFS
clusters,
but
maybe
you
want
to
utilize
some
of
its
nodes,
then,
on
this
notes
you
can
deploy
an
NFS
gateway
and
then
put
one
of
the
root
of
HDFS
to
route
to
the
off
to
the
local
and
take
the
NFS
and
take
this
as
a
local
column
and
X
is
utilizing.
A
Utilizing
the
loot,
so
different
storage
will
have
different
routes
and
also
it
needs
your
deployment
on
different
notes,
because
you
need
you,
the
cave.
It
should
be
deployed
on
each
of
the
nodes
and
also
should
provide
some
different
services
to
it
and
a
lot
of
configuring
configurating
as
well.
So
the
process
of
configuration
will
be
a
little
bit
complicated,
but
now
we
have
CSI
the
auto
identification
mechanisms
and
mounting
mechanisms.
It
will
be
much
easier,
so
I
think
in
kubernetes,
which
is
quite
suitable.
A
B
B
A
B
A
So
what
is
the
benefits
of
Chilton
I
say
this
as
the
storage
on
it,
because
it's
he
has
three
replicas
and
also
its
reliable
during
its
existence,
and
it
has
very
strong
enforcement.
Once
you
store
this
data,
then
you
can
ensure
the
consistency,
it's
impossible,
that
you
write
it
in
this
way
and
then
it's
reading
is
different
and
the
speed
of
reading
is
very
high.
A
It
also
able
to
route
different
paths
in
on
your
local.
Of
course,
you
you
can
route
different
paths,
so
it's
suitable
for
different
users
because
they
have
different
authentication
authentication
make
they
may
see
different
directory.
So
it's
able
to
route
different
users
to
different
different
place.
B
A
Also,
can
utilize
mounted
nodes
mounting
and
read
and
write
the
mounted
nodes,
ridden
writes,
has
one
limit
you
and
you
cannot
use
writing
the
same
file
at
the
same
time.
It's
not
allowed.
As
for
snapshot
in
you
can
your
reside,
you
can
do
recite.
Dynamic
recite
is
not
supported
here,
because
the
snapshot
is
not
spot
here.
B
A
A
A
A
B
A
A
A
B
A
Has
several
NFS
employ
did
have
any
problems.
For
example,
if
you
have
100
volumes,
we
haven't
producing
a
larger
scale
because
for
some
of
the
applications
and
we
are
different
to
control
which
what
is
utilizing
wood
volumes
in
these
occasions?
What
should
we
do?
I
think
this
is
the
practical
usage
it's
difficult
to
validate
in
a
moment,
I
think
the
volume
may
be
able
to
share
between
different
ports,
I
think
it's
okay.