►
Description
Don’t miss out! Join us at our upcoming events: EnvoyCon Virtual on October 15 and KubeCon + CloudNativeCon North America 2020 Virtual from November 17-20. Learn more at https://kubecon.io. The conferences feature presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Elephant on Wheels: Petabyte-scale AI @ LinkedIn - Cong Gu & Abin Shahab, LinkedIn
Kubernetes has flourished at LinkedIn for AI workloads. It started as a proof of concept for Jupyter notebooks, and now it has become a key infrastructure for model training and model serving. LinkedIn AI has been traditionally Hadoop/YARN based, and its Hadoop data lake is one of the worlds largest. To allow AI and non-AI workloads to securely access HDFS, a scalable, secure, open-source integration with HDFS Kerberos called Kube2Hadoop was built. This enables AI modelers at LinkedIn to use data securely in their model exploration and training with KubeFlow components such as the mpi-operator. LinkedIn’s infra teams are also prototyping a multilevel scheduler on top of Kubernetes and YARN clusters on the cloud, which can intelligently route jobs to multiple clusters and can facilitate workflows across Kubernetes and YARN clusters.
https://sched.co/Zepx
A
A
A
A
Our
stack
is
based
on
hadoop
users,
submit
jobs
via
azkaban
to
spark
or
tony,
and
then
they
run
on
yarn.
We
run
one
of
the
largest
yarn
clusters
in
the
world
with
more
than
6000
nodes
and
with
a
very
high
growth
rate
in
the
cluster
size
side
by
side
we
have
a
kubernetes
cluster
that
users
use
to
run
deep
learning
jobs
that
access
our
petabyte
scale
data
store.
B
Thank
you
robin,
as
abin
has
mentioned
in
the
previous
slide,
linkedin
has
traditionally
been
using
hadoop
as
its
compute
platform,
and
we
operate
one
of
the
largest
hadoop
data
lake
in
the
world
to
enable
linkedin
jobs
running
on
kubernetes
means.
We
first
need
to
tackle
the
problem
of
accessing
hdfs
data
from
kubernetes.
B
B
B
Now
any
further
communication
between
the
user
and
the
name
node
no
longer
need
to
authenticate
with
the
kerberos
server.
This
really
helps
hadoop
data
access
scale,
as
you
can
imagine.
If
every
request
requires
authentication
against
kerberos,
the
authentication
process
would
soon
become
the
bottleneck
for
the
entire
system.
B
The
kerberos
server
also
serves
many
other
authentication
requests
other
than
user
data
access
request.
The
hadoop
delegation
token
by
default,
has
a
lifespan
of
one
day
and
can
be
renewed
for
up
to
seven
days.
Ai
modelers
at
linkedin
uses
hdfs
data
to
train
their
models.
They
gain
access
to
data
through
hdfs,
using
user
account
or
headless
accounts.
B
B
B
B
B
B
Now
to
understand
cube2
hadoop
more,
let's
look
at
a
typical
workflow,
a
user
will
log
on
to
some
gateway
machine
using
their
password,
and
they
will
get
a
certificate
using
that
certificate.
The
user
can
submit
a
job
to
kubernetes
cluster
once
the
job
is
submitted,
the
worker
pods
will
be
launched
and
now
that
the
order
pods
have
been
launched,
the
init
containers
in
those
pods
will
start
to
send
the
request
to
a
duke
token
service
to
get
tokens.
B
The
hadoop
token
service
then
act
as
a
hadoop
super
user
and
will
contact
name
node
to
fetch
delegation
token,
in
step
in
step
6.
After
the
token
is
returned
to
dna
containers,
any
container
will
mount
those
tokens
locally
onto
the
pod.
In
the
shared
volume
in
step
7,
all
the
workers
will
be
able
to
access
hdfs
data
securely.
B
B
B
B
B
B
B
B
B
A
A
A
A
Account
next,
this
request
will
be
converted
by
the
tf
job
operator
into
a
pod
submission
request
and
go
back
to
the
submission
request
pipeline
of
the
kubernetes
master.
In
that
submission
request
pipeline,
the
ig
decorator
will
again
intercept
it
and
decorate
it
with
the
end
users
ldap
id
in
this
way.
It
id
decorator
ensures
that
every
request
and
every
resource
ends
up
with
the
the
creating
end
users
ldap
id,
regardless
of
what
service
account,
creates
it.
A
A
Kubernetes
strength
is
support
for
multiple
crd
to
scale,
support
for
all
the
different
crds
in
use
in
our
cluster.
We
wanted
to
make
sure
that
we
don't
have
to
change
id
decorator
code.
Every
time
we
introduce
a
new
custom
type
to
kubernetes
and
we
can
keep
using
id
decorator
for
any
new
custom
type.
We
import
into
kubernetes
to
ski
to
support
this
ig
decorator
uses
kubernetes
json
path,
we've
modified
it
a
little
bit
the
jsonplath
implementation
and
it
uses
that
to
decorate
the
pod
template
in
a
type
agnostic
way.
A
So
on
this
example
here
on
the
left,
there's
a
tf
job
that
gets
submitted
to
kubernetes,
and
that
has
a
particular
that
has
a
pod
template
inside
it
and
then
on.
The
right
is
a
specif
kind
of
a
specification
in
json
of
in
yaml
of
what
json
path
should
be
updated
to
insert
the
user's
ldap
id
and
therefore
id
decorator
will
use
this
configuration
to
find
under
the
pod
template
the
metadata
annotation
section
of
the
tf
job
and
decorate
the
user's
ldap
id.
A
There
now
a
larger
question
is:
why
do
we
need
cube
to
hadoop
at
all?
Kubernetes
has
secrets,
and
we
can
use
those
to
directly
mount
a
delegation
token
that
the
user
creates.
Why
do
we
go
through
the
circuitous
route
of
having
a
service
that
sends
the
tokens
back
to
the
use
back
to
the
jobs
that
are
running?
A
The
shortcoming
of
using
secrets
is
that
we
have
to
now
configure
very
fine-grained
our
back
rules
for
every
user
in
kubernetes
or
at
every
user,
on
the
namespace
to
make
sure
that
no
user
can
steal
another
user's
delegation,
token
and
impersonate
them
also.
We
need
to
make
sure
that
there
are
special
rules
for
administrators
who
could
see
delegation
tokens,
and
we
also
need
to
make
sure
that
the
there's
container
lifecycle,
hooks
or
or
culling
jobs
that
actually
go
ahead
and
clean
up
the
secrets
after
the
job
finishes.
A
B
B
Therefore,
we
purposefully
set
a
15
seconds
delay
before
the
first
request
from
the
init
container
to
a
do
token
service
to
account
for
the
propagation
time.
This
temporary
fix,
however,
was
not
satisfying,
as
our
customer
have
been
enjoying
the
super
fast
launch,
speed
of
their
kubernetes
jobs,
and
this
takes
it
away.
B
B
That
is
the
token
request
and
token
review
api
that
we've
shown
using
tokens
instead
of
ip
addresses
as
authentication,
we
no
longer
need
to
bind
ourselves
to
pod
network,
and
thus
we
can
support,
hosted
network
jobs,
and
since
we
won't
face
the
ip
address
propagation
issue,
there's
no
ip
address
at
all.
We
would
we
could
potentially
speed
up
our
init
container
by
a
very
big
margin.
B
Finally,
there
could
be
some
potential
future
improvements
for
cube
to
hadoop,
even
though
we
haven't
faced
this
issue.
Yet
this
is
definitely
a
must.
If
you
want
to
scale,
we
hope
to
let
chief
to
distribute
the
delegation
tokens
instead
of
every
single
worker
fetching
their
own
delegation
token,
you
can
imagine
with
a
popular
deep
learning
job
you
can
create
up
to
hundreds,
if
not
thousands,
of
containers
for
a
single
job.