►
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2023 in Amsterdam, The Netherlands from April 17-21. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
On Data Locality in Kubernetes - Chen Wang, IBM Research & Shouwei Chen , Alluxio
While Kubernetes has made it exceptionally easy to deploy and scale data-intensive applications elastically, accessing data from cloud-native data sources (like AWS S3 or sometimes remote data warehouses) becomes more challenging. Platform engineers often have to copy data to optimize the I/O throughput, which is error-prone and time-consuming. As the Kubernetes ecosystem matures and becomes more efficient, this challenge gets more imperative to address, and different attempts are being made to bring back data locality and influence workload scheduling. In this talk, we will discuss the pros and cons of different approaches to emulate or introduce data locality in Kubernetes schedulers. We believe this will become crucial for Kubernetes to achieve higher efficiency in the near future for data-intensive workloads.
A
Yeah
good
afternoon,
everyone
I
really
like
happy
to
stand
here
to
share
our
topic:
how
to
bring
the
daughter
locality
in
the
kubernetes
cluster
and
I'm.
Sorry,
actually,
as
a
interest
introduced
like
I'm,
the
Comm,
International
and
I
started
working
with
this
project
since
2016,
and
nowadays,
I
also
take
some
responsibility
to
do
the
product
manager
to
build
a
roadmap
for
this
project.
B
So
hello,
everyone
I'm
very
excited
to
be
here
and
then
today,
I'm
Chen
Wang
from
I'm
a
Staff
research
scientist
from
IBM
research
and,
on
my
part,
I
contribute
to
open
source
accumulated
a
lot
involved.
I've
been
involved
in
like
kubernetes
Auto,
scalers,
kubernetes,
scheduler
plugins
and
some
sustainable
projects
that
we'll
talk
on
Thursday
called
Kepler
and
then,
besides
that
I
have
my
my
research
interests.
They're
mainly
lie
in
two
size.
One
is
trying
to
use
machine
learning
to
improve
the
resource,
management
and
system
management
on
kubernetes.
B
So
I
will
start
with
a
brief
background,
and
what
is
why
we
started
this
problem
as
a
research
problem,
so
kubernetes
is
actually
a
natural
fit
for
running
Ai
and
ml
workloads
for
a
lot
of
reasons,
so
it
can
help
you
to
scale
to
meet
the
resource
requirements
for
your
an
ml
training
and
production
in
Virtual
workload.
It
gives
you
better
support
on
continuous
development,
which
is
by
nature
required
by
Ian
and
ml
workload,
and
then
it
helps
you
provide
an
abstraction
layer
that
allows
data
scientists
to
access
Services
visual
without
worrying
the
underlying
infrastructure.
B
It
provides
you
high
availability
field
over
protections
that
will
impact
your
SOA
and
resilience
problem.
It
allows
the
AIML
operations
to
operate
across
public
clouds,
private
clouds,
on-premise
infrastructure
and
also
include
secure
air
gap
locations,
and
it
also
gave
you
a
consistent
platform
for
different
stages
of
your
AI
and
ml
workload,
including
pre-processing
training
and
production
influence,
and
in
addition,
we
have
all
types
of
Open
Source
tools
available
on
to
be
used
on.
Kubernetes
include,
including,
like
Argo
workflow
management,
kubeflow
Ray
and
torch
acts
Etc.
B
So
when
you
start
running
a
pipeline
of
data
analytics
workload
or
a
machine
learning
pipelines
on
kubernetes,
there
are
a
lot
of
questions
coming
into
your
mind
like.
Where
do
we
store
our
data
permanently?
Where
can
we
cache
our
data
in
kubernetes
temporarily,
and
then,
where
do
we
place
paths
given
the
data
locality,
we
know
in
clusters
and
then
how
to
distribute
and
load
data
fast
enough.
B
So
we
improve
the
overall
cluster
resource
utilization
and
how
we
scale
resources
upon
like
data
volumes,
when
the
data
volume
becomes
really
big,
how
we
scale
resources
upon
that
and
then
what
apis?
We
should
use
to
get
those
data
into
our
Computing
paths
right
and
how
to
guarantee
the
persistence
and
resilience
of
your
data
when,
like,
for
example,
you're
Computing,
a
puzzled
completely
gone,
and
how
do
we
forward
data
between
different
stages
of
your
pipeline
and
then
how
do
we
manage
the
whole
life
cycle
of
data
in
kubernetes?
B
So
in
kubernetes
we
actually
don't
have
a
lot
of
support
of
those
questions
and
we
don't
either
have
the
isolation
and
the
data
security
guaranteed
amount
tenants
in
the
same
cluster.
So
we
will
I
will
cover
several
questions
of
those
and
then
show
we
will
present
you
what's
available
in
open
source
community.
B
So
we
start
all
the
study
based
on
building
a
simple
Benchmark
workflow
workload
called
we
built
the
Benchmark
based
on
the
existing
monolithic,
Tesseract
application.
So
Tesseract
is
an
open
source
optical
character,
recognition
engine.
It's
used
to
it's
a
very
popular
open
source
application,
but
it's
monolithic.
We
try
to
decompose
it
into
different
microservices
into
two
types
of
benchmarks,
and
then
we
did
all
those
evaluations
and
experiments
based
on
this.
This
benchmark.
B
So
as
I
mentioned,
tensor
Reiki
is
a
complete
workflow
of
machine
learning
pipeline.
It
includes
different
promises,
like
it
read
the
the
chief
Imaging
and
it
does
the
record
recognition
stage
with
finding
the
lines
and
running
all
those
machine
learning
inference
to
recognize
words
and
detect
the
paragraphs.
And
then
we
build
two
benchmarks.
B
So
so,
when
you
want
to
load
the
data
to
your
Computing
path,
there
are
several
options
intuitively.
So
you
can
directly
read
the
data
from
cloud.
Object,
storage
on
FS
server
or
you
can
mount
the
data
in
the
local
volume
for
the
from
the
disk
or
from
memory,
and
then
we
basically
evaluate
the
speed
to
get
the
data,
and
then
those
are
some
results.
Like
we
read
from
local
memory.
Local
disk
from
local,
persistent
volumes
I'll
be
loaded
from
an
advice
server
and
we
can
see
over
the
different
size
of
the
data.
B
The
maximum
speed
up.
We
can
get
that
the
bandwidth
throughput
Improvement
we
can
get
over
the
remote
radiant
right
can
can
be
improved
up
to
4.4,
41
bandwidth,
throughput
increase,
so
meaning
you
you
need.
You
need
much
less
time
to
load
the
data
when
the
data
is
already
in
your
local
volume
and
then
we
also
compare
the
local
disk
local
memory,
reading
right
versus
the
object,
storage
and
then
for
different
sizes.
You
can
see
maximally,
we
can
get
almost
10x
performance,
improved
and
minimum.
B
So
the
second
question:
if
we
assume
we
can
load
all
the
data
in
the
cluster,
do
the
pre-fetching
and
then
what
we
can
do
about
those
data
when,
for
example,
at
the
platform
layer
in
kubernetes,
so
where
we
put
schedule
our
place,
our
paths
in
kubernetes.
So
in
this
experiment
we
have
a
two
setups.
One
is
a
local
setup,
which
is
we
try
to
place
all
the
parts
using
the
data
on
the
single
node,
the
other.
Is
we
try
to
map
that
doing
some,
something
like
load
balancing
the
default?
B
Scheduler
will
do,
which
is
try
to
spread
the
part
across
noting
a
round
robbing
fashion,
and
then
at
this
time
we
just
want
to
take
a
look
at
the
I
o
time
it
takes,
and
then,
for
example,
we
did
some
experiments
on
one
page,
input
or
10
page
input
for
both
the
cross
cross
grain
Benchmark
and
find
green
Benchmark.
And
then
you
can
see
the
the
maximum
Improvement
can
up
to
five
times.
B
And
then
this
experiment
shows
the
end-to-end
processing
time
of
the
whole
workflow
and
then
for
the
cross
green
Benchmark.
We
can
see
the
maximum
input
can
be
four
times
if
you
try
to
consolidate
all
those
parts
using
the
data
on
the
same
node.
And
then,
if
you
develop
some
data,
Centric
Solutions,
meaning
you
try
to
place
a
pause
to
reduce
the
internal
node
data
movement.
Then
the
Improvement
is
also
like,
like
around
four
times
or
three
times,
at
least
so.
B
Next,
when
you,
when
you
run
a
very
large
workflows
in
your
clusters,
you
need
to
transfer
data.
There
are
a
lot
of
intermediate
data
between
stages.
You
want
to
get
the
output
of
the
state
stage
one
and
forward
the
data
to
the
the
to
stage
two
and
then
how
can
you
forward
data
between
stages
in
workflows
and
pipelines
and
similarly
the
The
Simple
Solution?
Is
we
read
and
write
to
a
remote,
remote
storage
and
then
this
serves
as
our
Baseline,
showing
the
NFS
read
and
write
and
the
other
solution?
B
Is
we
cache
the
data
locally
in
the
local
disk
or
local
memory,
and
then
we
can
see
for
different
sizes.
The
data
as
the
data
size
get
larger.
The
speed
up
can
goes
up
to
1.4
times
and
then,
if
we
do
something
like
direct
data
forwarding,
so
we
don't
store
it.
We
just
enable
data
Communications
between
parts
and
then
the
the
speed
up
can
further
be
2.5
x
and
then,
of
course,
we
have
some
star.
We
started
like
if
you
want
to
enable
the
direct
data
forwarding.
B
So
you
you,
don't
use
a
lot
of
resources,
but
at
the
same
time
you
achieve
the
good
performance
for
data
forwarding,
and
for
this
we
build
a
framework
called
vloggy
and
if
you
want
to
know
more
details
check
out
this
publication,
so
the
end
result
of
enabling
the
direct
data
forwarding
for
different
patterns
of
data
transfers
like
one
two
one,
one
two
and
end
to
end
and
then
those
are
the
results
and
the
maximum
layer.
We
can
improve
the
speed
up
up
to,
for
example,
60
times
so.
B
Enabling
data
forwarding
in
in
the
pipeline
is
important.
Of
course,
it
will
significant
significantly
reduce
the
end-to-end
execution
time
of
their
whole
machine
learning
pipelines,
but
you
would
also
lose
the
Persistence
of
the
intermediate
data
once
the
part
is
down.
So
it's
very
good
direct
data
forwarding
is
very
good
for
stages
like
it
doesn't
take
a
long
time
for
for
computing
or
processing,
but
it
loads
a
large
amount
of
data.
So
then
I
will
pass
the
slides
to
show
away
and
he
will
introduce
more
available
open
source
tool
to
solve
those
problems.
A
A
Today,
because
actually
we
talked
about
the
locality
actually,
where
mostly
focus
on
the
catching
capability
from
the
LSU
yeah
turn.
Imagine
that
there
are
a
lot
of
different
problems.
How
can
we
bring
the
data
located
in
the
kubernetes
environment
and
I
will
try
to
answer
most
of
them
in
my
deep
dive?
A
First
of
all,
I
want
to
give
some
basic
introduction.
What
is
the
performance
bottleneck
in
the
machine
learning
data
processing?
Actually,
in
general,
this
is
very
typical
machine
learning
data
processing
pipeline.
Actually,
you
will
load
the
data
from
the
under
layer
storage.
It
can
be
the
cloud
storage,
on-premise,
storage,
Object,
Store
and
even
the
local
disk.
Actually,
you
want
to
load
this
data
into
CPU
memory
and
afterwards
you
want
to
do
some
pre-processing,
like
you,
want
to
resize
your
image,
and
you
want
to
clean
this
kind
of
data
Then
I.
A
Then
you
feed
the
discount
data
from
the
CPU
memory
to
the
GPU
memory
or
TPU
memory
to
do
the
folder
training
with
the
machine
learning
algorithm
and
but
actually
there
are
several
points
here,
the
first
of
all
when
you
load
the
data
from
the
under
layer
storage
to
your
memory.
Actually
they
have
a
lot
of
problems
called
Fetch
store.
A
Most
of
the
time
is
IO
bound
because
actually
you
storage,
you
cannot
provide
enough
value
bandwidth
to
the
CPU
memory
and
afterwards
you
will
see
like
you
will
see
the
PayPal
or
store
actually
is
a
CPU
performance
bound
and
afterward.
You
will
see,
because
your
algorithm,
or
because
of
your
correct
characteristic
of
your
workload,
you
always
see
a
CPU
bound
there
and
if
you
want
to
see
more
analysis,
actually
there
is
a
one
very
good
resonance
on
the
layer
from
the
MSR.
A
Actually,
they
give
a
lot
more
explanation
there,
and
this
is
also
very
typical,
like
people,
how
do
how
they
do
like
a
machine
learning
training
in
their
environment.
Actually,
from
the
left
side
you
can
see
there
can
be
like
Object,
Store
disk
or
even
like
a
very
traditional
data
house
data
warehouse
with
hdfs
cluster
and
then
right
hand
because
of
the
benefit
of
the
kubernetes
allows
you
to
do.
The
multi-tenancy
find
granularity
control
of
the
results.
A
Actually,
people
always
doing
like
a
machine
learning,
training
in
the
kubernetes
environment
and
previous,
like
you,
will
directly
access
the
data
from
the
object
store.
You
can
copy.
Do
the
route
data
replication
from
the
storage
system
to
the
training
data,
which
will
involve
a
lot
of
manual
work,
including
the
replication
and
data
validation,
and
also
you
have
to
guarantee
the
data
prone
at
this
process.
A
Actually,
it
caused
a
lot
of
difficult
problems
there,
because
there's
no
data
scientists
rarely
want
to
deal
with
this
kind
of
dirty
work
from
the
data
Pipeline
and
also
it's
really
delays
the
delivery
of
the
product
and
another
problem
is
actually
as
Tim
mentioned
before.
If
you
direct
a
query
from
the
under
layer,
object,
storage
or
this
kind
of
fast
system
actually
performance
is
very
bad.
A
It's
not
only
about
iOS
throughput
between
storage
system
and
the
training
cluster
actually
is
also
about
the
metadata
operation,
like
listen
schedule,
so
it's
also
very
like
slow
from
this
kind
of
system.
So
actually,
when
I
think
about
yeah,
actually,
because
there's
no
data
locality
guarantee
from
this
kind
of
architecture,
so
actually
we
have
to
bring
back.
Otherwise,
those
GPU
utilization
is
always
very
low.
A
It
will
be,
it
can
be
like
10
or
20
percent
from
the
real
production
environment,
and
actually
it's
a
well
costs
a
lot
of
waste
because
actually
you
know
like
GPU
single
node
can
be
30
or
60
dollars
per
hour.
Actually,
you
just
utilize
one-fifth
of
its
resource.
Actually,
it's
a
big
waste
of
money.
There.
A
And
afterwards,
like
we
come
up
with
ideas,
so
let's
see
we
have
to
bring
this
intermediate
layer
into
this
kind
of
architecture.
So
the
first
of
all,
we
are
thinking
about
what
kind
of
interface
should
we
provide
to
this
kind
of
company
from
one?
No
matter
is
touch
tensorflow
or
even
like
internet.
This
is
the
mx90
this
kind
of
framework.
So
we
do
a
lot
of
investigation
and
we
found
like
no
matter
which
data
loader
you
are
using.
They
always
have
some
thing
called.
A
They
will
use
local
file
system
actually
because
a
lot
of
people
when
they
do
the
development.
Actually,
they
were
not
considered
to
use
like
data
loader
competitive
with
the
cloud
storage
or
data
loader
compared
to
the
hdf
as
this
kind
of
file
system,
but
they
always
use
the
posix
interface.
So
first
of
the
answer
to
it
is
actually
we
also
developed
the
process
interface
with
the
analysis
fields,
to
provide
the
best
capability
to
solve
this
kind
of
training
model
and
the
second
problem
how
we
can
bring
the
data
locality
back
into
this
architecture.
A
First
of
all,
we
will
have
the
cluster
a
cluster
level
locality
at
the
left
side,
which
means
you
can
ignore.
Where
is
under
storage
ATS?
This
can
be
stored
like
a
cloud,
storage
or
Sim
origin
cloud
storage
at
a
different
region,
and
also
it
can
you
can
access
data
from
the
cloud
to
your
own
promise
data
center,
which
even
a
geodesfield
data
center,
which
have
very
high
latency.
This
is
the
level
part
actually
is
guaranteed
this
part
of
the
locality,
the
second
locality
actually
is.
A
We
want
to
also
bring
the
locality
back
to
another
level,
because
you
know,
like
even
you,
bring
back
this
kind
of
locality
from
the
remote
storage
to
the
local
storage.
You
still
want
to
see
like
it's
a
lot
of
performance
difference
between
your
access,
local
compared
to
the
remote
like
distributed
file
system.
A
So
actually,
the
second
part
is
way
well
co-located
or
I'll
actual
Fields
part,
together
with
the
machine
learning
training
port
to
make
sure
like
we
can
catch
the
metadata
as
a
larger
field
site
and
also
cache
the
data
from
the
electric
client
side
to
serve
the
best
performance
for
the
machine.
Learning
training
and
another
thing
is
like
because
actually,
where
I
catching
layer,
where
the
Elijah
itself
will
not
never
guarantee
the
persistence
for
the
data,
which
means,
when
you
write
the
data
back
or
you
access
the
data,
we
will
not
guarantee
like.
A
You
were
not
lost
the
data
because
we
always
do
the
invection.
If
the
calcium
space
is
insufficient.
So
at
the
under
layer
we
will
write
back
the
data
to
the
under
layer
storage.
It
can
be
the
cloud,
storage
or
on-premise
storage
system
to
guarantee
this
kind
of
persistent,
and
when
you
get
the
performance,
the
temporary
data
is
stored
in
a
larger
space.
A
The
big
problem
here
is
like,
although
we
have
this
kind,
this
kind
of
catching
system,
but
you
know
like,
if
you
don't
know
any
access
pattern
of
your
machine
learning
processing,
you
always
to
say
like
what.
What
is
the
first
time,
the
first
time
always
very
slow.
When
you
do
the
fourth
airport,
it
can
take
forever
to
do
the
data
from
the
cloud
storage
or
the
remote
Object
Store,
and
we
say
like
okay,
we
provide
a
way
for
you
to
do
this
kind
of
perfect.
We
call
distribute
load
in
our
system.
A
First
of
all,
you
can
initialize
the
distribute
load
command
from
the
electric
client
to
the
electric
Master.
The
allergic
Master
were
aware
of.
Okay
I
need
this
kind
of
data
from
the
under
file
system,
so
we'll
load
the
data
first
from
the
cloud
storage
into
a
larger
worker,
so
which
means
it's
overcome.
The
first
remove
fetch
from
the
remote
data
center
or
the
cloud
storage.
Then,
second,
actually,
a
large
refuse,
a
port
where
passive
Vector
data
went
I
mean
when
the
machine
learning
Port
will
want
to
access
discount
data.
A
It
will
catch
the
metadata
at
a
larger
field.
Spot
and
also
at
the
same
time,
it
will
also
cache
the
data.
After
the
first
time
you
access
the
data
from
the
machine
learning,
job
and
afterward.
We
will
feed
the
data
into
the
machine
learning
training
part
and
this
data,
where
residents
in
the
CPU
memory
or
the
GPU
memory.
A
Okay,
the
last
problem
we
want
to
answer
because
actually,
my
main
job
actually
itself
has
a
lifestyle
command.
A
life
cycle
acrostic
characteristics.
Actually,
we
want
to
see
how
we
can
manage
this
kind
of
less
life
cycle
of
the
data
in
a
kubernetes
environment
at
the
beginning.
Actually,
we
use
the
sidecar
mode
to
manage
all
this
kind
of
data,
but
afterwards
we
realized
it's
not
the
best
way
to
do
it.
So
we
adopted
the
CSI,
which
means
the
container
storage
interface,
to
provide
a
better
lifecycle
Management
in
the
kubernetes,
the
problem
of
the
cyclical
mode.
A
Actually,
we
will
launch
the
I
should
feel
support
from
the
node
server
when
the
node
server
have
some
problem.
Actually,
you
have
to
restart
your
application
port
and
also
same
as
allergic
field
spot
to
restart
the
whole
service.
Actually,
it's
not
so
friendly,
no
matter
for
your
daily
maintenance.
Also
it's
about
like
when
you
want
to
upgrade
this
kind
of
service
they
actually
they're
like
hard
binding.
So
actually
it's
not
referended
to
the
data
scientist,
but
afterwards
we
find
that
it's
better
ways
to
use
CSI.
A
When
you
launch
the,
when
you
launch
the
application
Port,
actually,
we
will
also
launch
the
iso
field
spot
from
the
CSI
component
at
the
first
time,
and
we
also
mount
the
TV
into
this
two
part
to
provide
the
data
in
the
local
machine
and
at
the
same
time,
if
you
launch
more
application
port
in
the
same
same
hosting
machine.
Actually,
the
only
thing
you
need
to
do
is
like
launch
more
application.
Part
with
mounting
this
kind
of
processing
volume
into
this
container.
A
Yeah
and
afterwards,
I
want
to
see
share
some
result.
What
will
happen
after
you
adopt
this
solution
in
your
production,
environment.
I
would
not
say
this
is
the
best
case
to
show
the
performance,
because
we
should
we
see
much
better
performance
in
the
environment,
but
this
is
very
typical,
like
average
use
cases
for
the
real
production
environment,
as
you
can
see
when
you
load
the
data
directly
from
the
object
store,
you
can
see
here.
Actually
it
has
very
low
GPU
usage,
the
maximum
GPU
usage.
A
In
this
kind
of
environment-
and
the
second
problem
is
like
previously,
you
have
to
do
the
data
replication
to
guarantee
the
performance
and
they
have
to
do
the
warm
warm
the
data
before
the
training.
Actually,
it's
actually,
when
you
launch
the
GPU
cluster,
you
have
to
pay
for
the
money.
Then,
nowadays
you
don't
need
to
launch
the
data
to
do
a
launch
this
cluster.
To
do
the
warm-up.
You
can
just
warm
up
the
data
and
launch
the
GPU
class
afterward.
E
So
I'm
curious
in
your
research,
which
was
very
thorough.
Incidentally,
have
you
looked
at
some
of
the
kernel?
The
Linux
features
like
cache
files
D,
for
example,
that
lets
you
cache
NFS
or
sift
data
locally,
to
see
if
something,
that's,
maybe
less
intrusive
or
less
complex
as
the
product
displayed
here,
but
still
gives
you
good
Returns
on
your
on
your
performance.
C
Oh
yeah,
that's
a
that's
a
very
good
question
and
yeah
right
now
we
haven't.
We
have
a
that's
a
good
research
question.
We
we
can
try
to
study
together
in
the
future.
If
you
are
interested
and
then
right
now
we
just
Implement
some
simple
stuff,
like
the
direct
data
forwarding
and
local
volume,
cash.
D
C
A
As
we
in
the
community
to
contribute,
contribute
constantly
also
gave
a
quick
answer
for
this
one.
Actually,
because
I
coordinated
itself,
don't
provide
data
locality
schedule,
which
means
like
maybe
you
can
provide
the
IP
for
the
hosting
machine,
and
the
Computing
framework
also
have
to
aware
of
this
kind
of
change
and
give
it
back
to
them
actually
previously
do
a
lot
of
work
to
enable
this
by
directory
the
difficult
parties.
So
you
also
have
to
push
into
the
Computing
framework
to
make
their
schedule
to
Avail
this
kind
of
change.