►
From YouTube: Enterprise Machine Learning on K8s: Lessons Learned and the Road... - Timothy Chen & Tristan Zajonc
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Enterprise Machine Learning on K8s: Lessons Learned and the Road Ahead - Timothy Chen & Tristan Zajonc, Cloudera
Kubernetes as a platform is being asked to support an ever increasing range of workloads, including machine learning and big data processing. These new workloads introduce challenges both for both end users and cluster administrators. Data scientists want the flexibility to run any workload and library they require, data engineers want to ensure the scalability and reliability of production workloads, and cluster administrators want to maintain governance and control over cluster resources. At Cloudera, we’ve built a machine learning platform on Kubernetes that seeks to balance these competing objectives. In this talk, we will share some of the key design choices we made, lessons learned supporting large enterprise customers, and our vision of the road ahead for machine learning and AI on Kubernetes.
To learn more: https://sched.co/GrW2
A
Today,
I
want
to
talk
about
enterprise,
machine
learning
on
kubernetes
and
particularly
a
little
bit
of
emphasis
on
the
enterprise
bit
I
suspect
many
of
you
have
been
to
some
of
the
other
talks
you're
just
just
talking
about
machine
learning
in
general,
on
kubernetes,
but
Tim
and
I
have
been
doing
a
lot
of
work
at
Cloudera
on
how
do
you
support
enterprise,
machine
learning
and
leveraging
emerging
technologies
such
as
kubernetes?
So
we
want
to
give
you
some
of
the
lessons
learned
from
our
initiatives
and
a
little
bit
of
a
peek
at
the
road
ahead.
A
B
Yeah,
my
name
is
Tim
recently
joined
cloud
era,
so
now
I'm
a
leading
technology
implementation
for
cloudy
right
machine
learning.
We
used
to
have
a
startup
called
hyper
pilots,
so
you
see
a
little
bit
of
stuff
how
we're
going
to
be
looking
to
bring
some
of
a
hybrid
pilot
work
into
cloud
era.
Mushroom
you've
heard
of
our
stuff
before,
but
it's
also
very
kubernetes
native,
and
I
was
also
working
on
missiles
and
drill
and
also
helped
worked
on
spark
and.
A
My
name
is
Tristan
I
play
the
role
of
CTO
for
our
machine
learning
business
unit
at
Cloudera.
I
was
formerly
the
founder
of
small
start-up
called
sense
like
Tim,
which
got
acquired
by
clutter
about
three
years
ago.
We
were
building
what
we
called
an
enterprise
data
science
platform.
We
were
using
kubernetes
very,
very,
very
early
on
and
heavily
leveraged
containers,
and
that
has
carried
over
to
some
of
the
larger
scale
work
that
we've
been
doing
at
clutter.
A
So
let
me
just
I
have
to
do
this
for
lawyers,
some
of
the
stuff
that
I'm
talking
about
is
is
future
product
related
and
some
of
the
things
we're
building.
So
obviously,
no
promises
we're
talking
here
as
technologists,
not
as
sort
of
product
people,
all
right,
clutter
and
kubernetes.
So
I
just
want
to
start
off
by
asking
a
question.
Who
here
knows
what
cloud
era
does?
A
Okay,
so
maybe
half
the
people
so
just
to
orient
the
rest
of
I
Clara.
We
say
we
built
we're
building
a
modern
platform
for
machine
learning
and
analytics
optimized
for
the
cloud
we
in
particular
our
focus
of
large
customers,
huge
amounts
of
data,
huge
amounts
of
compute.
Most
of
you,
particularly
being
interested
in
open
source
probably
know
us
as
the
leading
distributor
behind
Hadoop
and
the
Hadoop
big
data
ecosystem
clutter
and
a
few
other
vendors,
you
know,
are
the
key
forces
behind
that.
A
You
probably
don't
know
us
as
being
heavily
involved
in
kubernetes,
because
there's
the
whole
Hadoop
and
Apache
ecosystem,
which
is
historically
not
been
tightly
connected
with
the
whole
cloud
native
ecosystem.
That
is
changing
and
some
of
the
things
that
we're
going
to
talk
about
today
are
about
how
these
two
communities
really
are
complementing
one
another
and
can
deliver
value
to
two
in
both
directions.
So
how
are
we
using
kubernetes
at
Cloudera?
I
want
to
just
be
clear,
because
this
is
what
were
we
talking
about?
A
Some
of
the
lessons
learned
from
so
the
first
thing
is
we
use
kubernetes
within
what
we
call
our
cloud
era:
data
science
workbench
product?
This
is
a
product
that
is
gives
a
collaborative
data
science
experience.
It
is
powered
by
kubernetes.
It
is
built
100%
on
doctrine
and
kubernetes
behind
the
scenes.
It
basically
accelerates
the
entire
life
cycle
of
doing
data
science
and
machine
learning
work
from
exploration
to
production.
A
It
works
on
small
data,
so
you
have
standalone
R
in
Python,
it
works,
it
can
connect
any
data
source
and
particularly,
we
have
a
Big
Data
Platform,
the
you
know,
the
Hadoop
platform
and
the
data
science
workbench
product
is
really
a
gateway
for
data
scientists
into
that
large-scale
data
engineering
scale
out
computes
tour,
äj--
platform
that
you
know
the
people,
people
that
here
know
the
Hadoop
ecosystem.
That's
probably
what
you
identify
it
with
so
sits
side
by
side
with
with
the
rest
of
our
stack,
but
it's
really
just
a
kind
of
a
gateway.
A
Small
right
couple
nodes
you
know,
gives
you
a
little
bit
of
a
workbench
environment
for
your
data
science
teams.
So
we've
been
shipping
this.
This
was
the
basically
what
I
was
building
at
cents.
When
we
were
a
startup,
it
became
clutter
data
science,
workbench
and
we've
done
a
lot
around
enterprise,
integration
and
security,
but
it's
really
a
gateway
in
to
the
Hadoop
environment
or
the
large-scale
sort
of
data
platform
environment.
A
So
last
week-
and
this
is
what
Kim
is
going
to
be
showing
you
some
of
the
technology
behind
about
a
week
ago-
clutter
announced
a
preview
for
sort
of
the
next
generation,
more
cloud
native
version
of
our
machine
learning
platform.
So
the
key
differentiating
factor
here
is
that
we've
taken
the
wheel.
We
enable
the
end-to-end
workflow
for
machine
learning
directly
on
kubernetes,
including
the
scale
out
compute,
namely
distributed
computing
with
spark.
A
So
this
is
a
product
that
you
can
install
natively
onto
you
know,
EK
SaaS
gke,
you
know,
OpenShift
has
connectivity
your
primary
Kuster,
but
is
in
a
separated
computing
storage
environment
where
you
have
scale-out
computing
on
kubernetes
that
can
include
you
know
your
standalone
R
in
Python,
tensorflow,
PI,
torch,
etc,
but
also
distributed
lot.
You
know,
frameworks
that
are
core
to
the
traditional
platform
or
the
hood.
Traditional
platform,
namely,
you
know
spark,
is
one
of
the
most
important
things.
A
So
that's
it
on
the
sort
of
like
I'm
just
trying
to
orient
you
in
terms
of
what
we
are
building
and
I
wanted
to
talk
to
you
a
little
bit
about
some
of
the
kind
of
high-level
lessons
learned
around
doing
machine
learning
at
scale
within
the
enterprise
leveraging
kubernetes
and
then
allow
Tim
to
talk
a
little
bit
about.
What's
going
on
behind
the
scenes,
how
do
we
actually
use
kubernetes
and
what
are
the
some
of
the
emerging
capabilities
that
exist
within
the
open
source
ecosystem?
A
So
some
lessons
learned
I
have
very
limited
time,
so
I'm
just
going
to
give
you
two
high-level
lessons.
So
the
first
is
that
within
the
enterprise,
enterprise
ml
really
does
require
the
bigger
picture,
and
this
is
somewhat
self-serving
because
clutter
we
say
we,
you
know,
provide
the
bigger
picture,
but
it
is
really
true.
If
you
look
at
how
the
large
companies
do
it
and
when
I
mean
by
the
bigger
picture,
is
I
mean
any
machine
learning
application
isn't
just,
for
instance,
running
a
PI
torch
job
or
a
tensorflow
job
right.
A
It
fits
within
a
broader
eco
data
platform
ecosystem
that
companies
have
you
have
data.
That's
streaming
in,
you
need
to
do
large-scale
data
engineering.
You
do
want
to
do
machine
learning
on
that.
Do
predictive
predictions
and
advanced
analytics,
but
you're
also
driving
data
warehouses
right,
so
you
taking
those
predictive
product,
those
printed
outputs
and
you're
driving
a
data
warehouse
that
your
business
user
could
use.
You
have
operational
products
that
you're
trying
to
build
right
recommendation
engines
for
your
for
your
movies
or
your
cell
phone.
A
You
know
something
it's
something
on
your
cell
phone,
so
you
need
to
have
some
operational
component
and
the
key
thing
within
the
enterprise
is:
how
do
you
glue
that
all
together?
And
in
particular,
how
do
you
have
this?
A
common
security,
metadata
governance
story
behind
behind
the
scenes?
Now
that
doesn't
mean
you
need
to
all
go
to
one
vendor.
You
can
glue
it
together.
You
know
by
taking
best-of-breed
products,
but
the
general
point
here
is
that
you
do
need
to
fit
enterprise
ml
within
a
broader
ecosystem.
So
you
don't,
you
know
I!
A
Think
of
like
oh,
it's
stupid
or
notebooks.
It's
not
Jupiter
notebooks
within
the
enterprise
environment.
Jupiter
notebooks
are
a
great
part
of
the
ecosystem,
but
that's
not
the
whole
story
for
doing
enterprise
ml
at
scale.
So
one
way,
so
what
is
required
from
a
technology
perspective?
One
way
to
look
at
it
is
look
at
what
companies
are
doing
and.
A
I'm,
sorry
I'm
making
sure
my
time
is
okay,
so
one
way
to
do
it
is
to
look
at
what
companies
are
doing
so
here's
Ebers,
you
know
I,
had
a
blog
post
on
their
machine
learning
platform,
and
this
is
their
diagram
that
they
have
and
you
can
see
it's
basically
the
diagram
that
I
put
before
right,
there's
Kafka
in
there
first,
which
is
part
of
the
Hadoop
ecosystem.
That's
for
streaming
data!
There's
a
data
Lake
sitting
there
there's
a
spark
job.
A
You
can
see
that's
doing
data
engineering
data,
prep
job,
it's
leading
into
a
batch
training
job,
so
they're
doing
that.
Who
knows
they
might
be
using
containers
there.
That
might
be
a
good
idea,
so
maybe
they're
using
containers
there
I
don't
know
meso
serve
or
kubernetes
they're,
then
going
into
an
operational
data
store
in
their
Conte
in
their
context.
That
is
Cassandra
and
then
they're
doing
two
things
they're
doing
real-time
predictions,
which
probably
is
a
micro
service
architecture
of
some
sort,
mesosphere
kubernetes
or
they're.
A
Doing
batch
prediction
where
again
they're
leveraging
SPARC
to
do
batch
prediction
at
scale
distributed
back
batch
prediction.
They
might
be
internal
to
SPARC,
using
machine
learning.
Library
like
tensorflow
Netflix,
has
a
very
similar
architecture
for
their
product
recommended
for
their
movie
recommendation
engine.
It's
pretty
much
honestly,
exactly
the
same
thing:
Kafka
SPARC,
you
know
some
machine
learning,
algorithms,
online
prediction,
online,
operational
databases,
Facebook,
is
building
a
lot
of
its
own,
but
it
has
a
similar
architecture
at
the
core.
They
have
a
core
infrastructure
servers,
compute
date
network.
A
On
top
of
that
they
have
their
data
platform,
which
includes
workload,
management,
deployment,
management,
machine
learning,
productivity
tool.
They
call
FB
learner,
and
on
top
of
that
they
have
these
open
source,
machine
learning
frameworks
or
libraries.
In
their
case,
they
are
heavily
invested
in
cafe
2
pi
torch,
onyx,
ok,
but
you
could
think
put
tents
or
floor
tensorflow
scikit-learn
up
there
too.
So
the
bottom
line
message
that
I
want
to
give.
Is
these
ecosystems?
Some
people
say?
Oh,
you
know
kubernetes,
you
know
it's
gonna,
kill
hadoop
or
something
like
that.
A
A
That's
still
a
major
major
part
of
the
broader
data
platform
and
then
on
top
it
is
being
augmented
by
modern
frameworks,
around
innovative
machine
learning,
so
tensorflow
PI
torch,
that
sort
of
tooling
and
a
modern
ml
stack
and
enterprise
ml
stack
is
really
spans
these
two
now
there
are
some
hard
choices
right
in
the
Hadoop
ecosystem.
We
have
yarn,
which
is
yet
another
resource
manager.
It's
our
resource
manager
behind
some
of
the
components
here,
like
spark
and
hive,
you
could
run
spark
potentially
on
kubernetes.
That
might
be
a
reasonable
idea
right.
A
All
right,
so
the
second
lesson
that
I
want
to
just
mention
is
you
really
need
to
focus
on
the
workflows
for
machine
learning,
not
the
frameworks.
The
frameworks
are
take
changing
of
my
frameworks
and
libraries.
I
mean
things
like
pi
torch,
tensorflow
spark
ray
whatever
your
scikit-learn
or
the
languages.
Python
are
Scala.
A
A
What
you
need
to
do
is
you
need
to,
and
data
scientists
want
to
use
all
of
these
tools
right,
it's
particularly
within
a
large
company,
and
so
what
you
really
need
to
do
from
a
platform
perspective
is
think
about
how
do
I
enable
the
work
flow
underneath
the
behind
the
scenes
and,
from
our
perspective,
there's
a
few
key
workflows
that
you
need
to
help
accelerate
and
help
data
science
to
be
more
productive
on
there's
a
development
workflow
which
is
interactive.
It's
you
want
it
to
behavior.
A
Like
your
laptop,
you
need
access
to
data
and
compute,
but
you
want
to
mutate
a
mute,
a
table
file
system,
pip,
install
package.
You
want
it
to
be
like
your
laptop,
like
a
development
environment,
you
want
a
principled
way
to
run
training
runs,
so
you
want
a
way
to
snapshot.
Your
train
training
code
submit
something
to
the
cluster,
run
it
for
a
day
or
two.
Do
the
training
compare
the
conformance?
So
you
need
something
around
ad
hoc
training.
You
need
something
around
a
sharing
and
collaboration,
so
you
need
something
some
people
like
to
share
things.
A
You
know
for
those
in
our
ecosystem.
You
might
be
familiar
with
shiny,
for
instance,
you
definitely
need
data
engineering,
so
you
want
a
batch
and
pipeline
system
that
could
includes
distributed
jobs
like
spark
and
then
of
course
you
want
to
get
to
production
and
be
able
to
run
a
model
so
within.
How
do
we
do
that
on
crew?
So
we
do
that
within
our
machine
learning
product
cutter,
our
machine
learning
running
on
top
of
kubernetes.
A
How
does
that
actually
work
for
your
you
for
Uecker
bananas,
people,
so
Clara
machine
learning
is
built
around
enabling
these
workflows
and
that's
the
abstraction
that
we
try
to
target.
So
we
have
one
not
what's
you
know
we
have
one
Operator,
we
kind
of
follow
the
operator
pattern.
We
have
four
core
CR
DS
I
put
it
in
quotes
because
we
don't
actually
use
series
currently,
but
that's
conceptually
what
they're
doing
the
first
is
for
sessions.
That's
interactive
and
mutating.
That's
things
like
Jupiter
notebook,
write,
an
interactive
environment,
development
firing.
A
Tim
will
show
you
some
of
that.
We
have
an
experiment,
which
is
how
do
we
go
from
source
to
image
to
run?
Give
you
an
immutable
version
that
you
can
kick
off
and
have
a
long
living
training
run.
We
have
a
job
which
is
a
scheduled
recurring
thing
and
we
have
a
model
which
is
an
online.
How
do
I
apply
something
as
a
restful
micro
service?
A
One
thing
that
I
want
to
point
out
here,
which
I
think
is
a
little
bit
cadential
controversial
is
we
do
not
have
long-lived
operators
for
any
particular
frame
framework,
so
we
do
not
have
a
TF
job.
We
don't
have
a
spark
job
or
anything
like
that.
We
try
to
let
the
frameworks
just
be
libraries
so
that
they
can
compose
well
into
these
higher-level
workflows.
Right,
so
you
can
have
a
spark
session,
you
can
explore
experiments,
mark
job
spark
model
and
the
libraries
themselves.
A
They
can
be
kubernetes
aware,
but
we
do
not
have
a
separate
long-lived
operator
for
every
framework
that
our
data
scientists
want
to
work
to
use,
and
that's
worked
quite
well
for
us.
It's
definitely
a
topic
worthy
of
questions.
Folks
have
them.
The
final
thing
is
kubernetes
is
not
exposed
to
data
scientists.
Our
goal
is
to
really
give
them
a
serverless
experience
and
I
love.
A
You
know
keeps
ETL,
but
honestly,
I
can
barely
get
some
data
scientist
to
love
get,
and
so
you
know
a
wall
of
Y,
a
mole
that
does
all
the
spark
configuration
is
not
what
most
higher-level
data
scientists
won.
So
we've
tried
to
build
a
system
where
you
can
get
to
kubernetes
if
you
absolutely
need
it,
but
for
the
end
data
scientists,
we
want
that
experience
to
be
as
service
as
possible
and
what
we
mean
by
that
is.
A
We
want
to
have
experience
and
we
don't
see
why
there's
any
reason
why
the
platform
can't
provide
an
experience.
That's,
like
you
know,
clutter
ml
run.
You
know
Python
train
dot,
PI,
you
snapshot
your
code,
you
put
it
up.
You
build
your
image,
install
your
dependencies
ship
around
your
dependencies
run
it
track.
There
were
results,
outputs
and
it's
done,
and
we
likewise
want
the
frameworks
to
have
a
symbol.
A
server
list
experience
so-
and
this
is
an
example
of
SPARC.
We
think
you
should
just
import
SPARC
import
from
PI
spark.
A
You
know,
sequel
imports,
work
session,
create
a
spark
session,
ideally
no
configuration
and
then
run
aid
in
this
case
a
distributed
map
over
a
function
f
and
that
should
spin
up
all
the
necessary
compute
resources.
It
should
make
sure
that
those
computers
is
have
the
dependencies
that
you've
installed
in
your
environment
and
all
that
should
be
ideally
completely
transparent
to
the
user,
and
that's
what
we've
been
trying
to
enable
so
there's
some
technology
behind.
How
do
we
do
this?
How
we're
leveraging
SPARC
and
I
will
turn
over
to
Tim?
B
All
right,
thanks
Tristan,
so
let's
move
ahead
and
talk
so
Tristan
gave
a
great
overview
about
the
high-level
lessons
learned
now.
I
want
to
actually
talk
a
little
bit
more
lower
level
details
about
some
of
the
technology
we're
using
and
how
actually
integrating.
It
also
show
you
what
is
cloud
era,
data,
science,
clutter
and
machinery.
How
does
that
look
like
running
SPARC
on
community
and
actually
running
on
manage
communities
overall,
so
we
talked
about
sparking
communities
and
I'm
I'm,
not
sure
how
familiar
you
all
are
with
spark
and
also
kubernetes.
So
maybe
good.
B
If
you
know
you
spark
before,
let
me
show
a
show
of
hands.
Okay,
so
it's
maybe
20%
all
right.
So
this
is
an
interesting.
How
many
of
you
actually
try
spark
on
kubernetes
yet
cause
about
to
sing
okay,
a
little
less
a
lot?
Okay,
obviously
maybe
15
20
people
all
right.
So
if
you
use
spark
on
Chrome
Nettie's,
you
definitely
understand
what
experience
looks
like
and
so
Sparkle
Corning's
it's
great
because
it
leverages
kubernetes
right.
It
has
the
kubernetes
operations,
things
that
you
like
about
communities
like
the
affinities.
B
You
know
toleration
x',
all
the
things
you
quest
limits
and
everything
you
build
tooling
around
kubernetes.
You
can
leverage
it
for
spark.
So
for
us,
sparking
cronies
is
great
because
we
can
leverage
our
existing
kubernetes
toolings
and
our
frameworks
and
stuff
we've
bills
for
kubernetes
and
bring
it
into
with
spark.
What's
most
interesting,
I.
Think
here
is
we
just
how
about
data
scientists?
They
want
to
use
various
tools
right,
like
pip,
install
of
Python
language
they
like
to
use.
This
could
be
tensor
flow.
B
This
could
be
scikit-learn,
it
could
be
anything
built
themselves
and
the
use
are
they
use
Scala
right,
there's
all
kinds
of
languages
right,
so
one
problem
when
challenge
for
you,
you
probably
will
see,
is
like
how
do
I
actually
ship
these
dependencies.
You
know
not
just
on
a
single
session
when
I
run
a
spark
job
right.
It's
distributed.
It's
gonna
run
containers
in
anywhere
in
the
nose
in
my
kubernetes
cluster.
B
How
do
I
ship
all
these
dependencies
to
distributed
environments,
and
it
makes
this
seamless
so
the
data
scientists
not
to
figure
out
oh
I-
need
actually
included
some
llamo
files
and
I
can
figure
out
all
the
things
and
build
my
own
docker
images
and
stuff
like
that.
How
do
I
make
it
seamless,
and
this
is
interactive
sessions?
B
This
is
not
like
pre
figured
job
and
long-running
batch
jobs
right
if
Barton
crew
neighs
we
talked
about
because
you're
running
equipment
Eddie's,
you
know
you're
running,
you
can
use
CPU
GPUs
and
have
auto
scaling
capabilities
in
the
cloud
and
overall
we
couldn't
talk
a
little
bit
about
how
we
get
to
the
improve
you,
ization
and
multi-tenancy,
but
because
of
bringing
into
the
same
resource
manager.
We
have
a
lot
more
opportunities
to
do
that.
All
right,
so
very
quick,
Sparkle
Cooney's
has
so
spark
used
to
have
no
native
integration.
B
We
brought
it
and
away
means
the
whole
community
of
spark
and
a
lot
of
kubernetes
people
helps
along
the
way
to
get
spark
have
native
integration.
That
means
you
don't
have
to
actually
care
where
screw
Burnett
ease
you
actually
configure
in
spark.
I
want
to
use
kubernetes
and
here's
my
spark
configuration.
It
will
just
talk
to
criminate
directly
and
run
it.
So
you
should
have
sat
alone
clusters.
B
You
know
spark
now
we
can
just
run
a
spark
directly
pointing
to
a
criminal's
api
and
it
will
run
just
executor
pods
for
you,
so
I
won't
go
into
too
much
details.
We
can
definitely
see
documentation
there
and
what's
great
about
it,
as
we
talked
about,
is
that
if
you
have
main
spaces
quotas,
our
backs
built
into
your
kubernetes
cluster
spark
can
leverage
that
too
and
I'll
show
you
a
little
bit
of
that
later,
all
right
still
so
sparkling
communities
in
cloud
era,
machine
machine
learning.
B
This
is
kind
of
how
little
deeper
dive
what
it
looks
like
right.
So
in
cloud
and
machine
learning
we
have,
we
have
what
we
call
engines.
So
an
engine
is
basically
where
we
are
backing
the
sessions,
it's
kind
of
like
a
trooper,
a
notebook.
When
you
create
a
dripper,
a
notebook,
you
have
interactive
session,
you
can
play
with
right.
So
for
every
session
create
we
create
an
engine.
This
engine
is
basically
kubernetes
pod
with
a
container
to
hassle.
Some
of
our
pre-built
dependencies
that
we
think
you
you'll
need
and
for
us
to
use.
B
So
you
need
to
actually
run
that
spark
driver
in
that
session
and
for
us
how
to
be
able
to
ship
dependencies
right
if
you
pip,
install
some
arbitrary
package
and
then
just
expect
spark
to
work
now.
How
does
that
work?
Well
for
us,
if
you
pip
install,
we
set
it
up
to
make
sure
that
your
pip,
installing
into
a
network
volume
that
we
can
able
to
ship
around
for
you
right.
So
this
could
be
enough.
It's
volume!
B
It
could
be
something
else,
but
if
you
set
it
up
the
right
way
and
you
have
a
home
directory
and
all
the
permissions
set
up
for
you
pip
install
into
that
get
your
spark
executor
x'
to
be
able
to
manage
your
all
the
necessary
information
to
find
those
packages
right.
So
you
can
just
propagate
that
all
around
to
different
executor
is
for
use
and
NSS
volumes.
You
know
this
is
user
permissions,
and
this
is
past,
setups
and
stuff
like
that.
B
So
we
do
it
at
all
for
you
behind
the
scenes,
but
one
important
piece
of
this
is
that
for
us
to
be
able
to
propagate
a
lot
of
information
from
the
main
session
to
the
executor
--zz,
you
need
to
able
we
use
something
called
executor,
paw
templates.
So
if
you
mean
following
the
sparklin
communities
work,
I
need
you
need
instead
of
telling
SPARC
here's
all
the
little
details
of
what
I
need
to
propagate
everywhere.
B
Give
me
a
template
almost
like
a
llamó
file,
but
this
is
populated
for
you
that
it
could
automatically
transfer
anything
in
this
pot
template
and
put
it
into
all
the
executives
at
at
launch.
So
this
could
be
labels
that
could
be
volumes
too
could
be
image,
pool
secrets.
It
could
be
a
whole
bunch
of
stuff
so
because
Coomer
is
adding
a
lot
more
fields
into
siamo.
B
It's
really
hard
to
add
one
one
by
one
and
spark,
because
you'd
imagine,
yeah
I
think
I
had
to
create
like
hundreds
little
knobs
and
spark,
and
it's
just
almost
impossible
so
having
a
generic
template
again
passed
around
is
actually
very
powerful
way
for
you
to
maintain
a
software
like
this.
An
important
part
of
this
is
that,
because
interactive
sessions
create
executors,
once
you've
killed
a
session,
everything
is
gone
right.
The
pot
is
gone.
Your
session
is
gone.
How
do
I
find
my
logging?
B
How
do
I
find
my
spark
metrics
right
for
spark
history
server
how
to
actually
debug
these
things?
So
we
set
up
flew
in
D
and
two
point
to
these
pause,
using
like
some
naming
scheme
and
label
that
we
can
find
these
pause
on
this
node
mounts
to
host
volumes
to
keep
the
shear
writing
the
logs.
Also,
the
spark
metrics
you
didn't
it
a
little
bit
gluing
to
because
spark
Mex
is
a
little
interesting,
but
you
need
it
actually
able
to
tail
it
out
into
this
serve
volume.
But
this
will
be
shipped
to
some
external
storage.
B
So
that's
very
quickly,
and
we
know
how
much
time,
but
that's
quickly
how
we
do
this
and
we
because
we're
using
kubernetes
we
actually
set
up
a
namespace
per
project.
So
that's
how
every
user
don't
just
create.
Whatever
party
wants
right,
we
give
you
a
quota
and
we
give
you
a
service
accounts
that
can
only
launch
executives
within
your
namespace.
So
you
can
go
crazy
in
your
namespace.
You
know
you
can
kill
your
own
secular
pause.
B
B
There's
all
this
much
more
detail
about
Kerberos
and
all
that
stuff
is
happening
all
right.
So
demo
I
want
to
show
you
what
this
kind
of
looks
like
in
our
clutter
and
machine
learning
and
I'll
make
sure
it
can
come
out
all
right.
So
let
me
start
here
so
see
this.
This
is
called
air
and
machine
learning
how
it
looks
like
it's
a
web
interface,
so
data
scientists
will
come
in
and
login
and
every
one
of
them
has
basically
projects
write.
A
project
is
a
shared
workspace
that
has
all
your
files
there.
B
You
can
collaboratively
edit
files
and
create
interactive
sessions
from
those
projects.
So
normally
you
create
a
project,
I
reap
recreated
one
and
you
have
your
files.
This
is
a
pice
spark
template
that
we
have.
You
can
there's
a
lot
of
things.
We
can
have
we
their
sessions,
experiments,
there's
small
deployments
and
jobs
and
stuff
like
that.
I
want
to
highlights
today
about
spark
on
kubernetes.
So
if
you
open
a
workbench,
a
workbench
is
basically
equivalent
of
a
super
and
awkward
we
just
talked
about.
B
You
have
an
interactive
session
that
you
can
watch
here,
so
you
can
choose
what
engine
you
want
this
engine.
We
have
our
own
engine
that
actually
has
a
lot
of
dependencies.
Like
you
know,
anaconda
or
scikit-learn
all
in
defaults
packages.
We
think
most
data
scientists
need
and
you
can
choose
what
language
you
wants,
because
that
set
us
all
language,
bindings
and
paths
and
stuff
like
that,
and
most
interesting
is
that
you
can
choose
how
much
resources
do
you
want
to
give
this
session?
B
If
you
have
GPUs
in
this
cluster,
it
automatically
detects
it
and
it
will
show
you
what
how
many
GPUs
do
you
want?
Oh
so,
for
this
session,
you
launch
the
session.
This
creates
this
interactive
session
for
you
and
you
can
so.
While
this
is
running,
let
me
explain
to
you
what
I
would
like
to
show
you
today.
So
how
do
you
get
dependence?
B
We
talked
about
how
we
actually
propagate
dependencies
across
executives
right
and
also,
how
do
we
actually
get
spark
on
crew
Nettie's
to
just
work,
so
this
example
is
tends
to
fall
in
spark,
so
this
is
pre
trained
models
but
feeding
tensorflow
some
image
nets,
images
to
do
inference
and
I'm
using
spark
to
just
spread
the
images
right
so
I
have
a
bunch
of
images,
I
create
our
G,
d's
and
batches
I,
spread
it
around
for
executor,
x'.
They
all
instantly
all
pip
installed
tensorflow.
B
When
they
don't
actually
do
that,
we
you
pip,
install
on
x,
fill
in
your
session.
It
automatically
propagates
tensor
field
package
to
all
the
executor
x'
for
you,
so
that
in
your
executor,
x'
when
you're
doing
an
inference,
it
army
automatically
picks
up
the
tensor
fold
dependency.
So
I
pip
install
tensor
full
already.
It
doesn't
come
with
our
default
engine,
but
normally
what
you
do
is
you
do?
Pip
install
tensor
fall
here
right.
This
installs
only
intercession,
but
let
me
just
run
this
really
quick.
B
But
what
do
you
tip?
What
will
happen
here
is,
basically
it
will
starts
to
download
image
nets.
Download.
The
images
needs
to
do
inference
on
and
start
to
spread
around
the
executor.
So
the
most
important
piece
here:
okay,
it's
right
here
right!
This
is
your
typical
PI
spark
code,
my
sure,
just
too
small.
This
is
your
typical
PI,
smart
code
right.
This
is
what's
just
christened.
Just
showed
you
right,
you
just
import.
Pi
spark
build
a
spark
session
and
run
it
and
we
will
in
in
our
engine.
B
We
do
we
have
that
something
called
in
its
process.
That's
actually
a
propagate
populates
default
configurations
for
your
session
and
we
we
populates
the
spark
kubernetes.
You
know
weird
if
I'm
the
API
server,
what
the
pond
names
are.
You
know
some
some
default
settings
that
this
data
sign
is
shouldn't
even
worry
about,
and
we
can
talk
about
later
about
the
how
even
more
advanced
optimizations
we
can
do
if
we
know
what
you're
running
on
right.
So
this
is
data
science
for
them
they
just
import,
spark
and
run
it
right.
B
Everything
is
already
set
up,
and
you
know
you
just
kind
of
just
finish
this
based
on
time.
We
can
really
go
through
so
much,
but
this
basically
goes
through
spreads
images
around
and
run
inference
on
them.
So
without
the
dependency
management
piece
of
mounting
executor
x',
let
me
actually
show
you
the
pause,
hopefully
he's
still
running,
without
actually
mounting
enough
s
pots
volume.
Since
your
HT
sector,
you
won't
even
even
run
the
example
because
this
example
knees
custom
packages
to
run
this
is
probably
a
long
time
ago.
So
I
have
executor
z--.
B
This
is
SPARC
executives
launched
for
me
for
this.
My
session
that's
running
right,
so
this
is
just
happening
behind
the
scenes.
I
have
exec
tourist,
launched
by
kubernetes
connected
to
this
current
client
Mo's
session,
able
to
give
out,
commands
and
get
the
output
back
and
show
you
right
here
into
your
session
right
here
right
so
to
the
data
scientist.
This
is
great
right.
I,
don't
have
to
figure
out
how
to
actually
configure
spark.
B
I
didn't
actually
figure
out
how
to
do
scale
outs,
because
this
is
kind
of
done
for
me,
of
course,
there's
a
lot
more
figuration
parameters
as
spark
is
pretty
manure
things
again
tune
right,
so
there's
there's
a
way
we're
going
to
give
you
a
bang
experiences,
spark
history,
server
and
all
the
tooling
and
logging
you
need
to
to
able
to
see
what's
happening,
but
so
that's
kind
of
how
tents
spark
on
khones.
Just
natively
integrates
right.
B
B
You
know
two
minutes
alright,
so
we're
gonna
go
for
this
really
quick!
So,
what's
what's
happening
right
now,
so
there's
there's
more
work
needed
on
the
spark
on
cronies
of
strings.
I
right.
There's,
there's
Kerberos
support,
that's
actually
finishing
in
spark
code
base,
dynamic
allocation
doesn't
work
yet
and
I'm
not
sure,
even
know
what
time
allocation
is.
That
means
it's
a
it's
a
auto-scaling,
that's
happening
in
a
spark
level
right.
B
Cumin
Hayes
has
auto
scaling,
but
if
you
have
to
tell
kubernetes
hey
look
at
this
metric,
if
it's
above
some
special
start
doing,
some
scaling
spark
has
a
lot
of
information
itself
right.
I
have
a
lot
more
tasks
queued
up,
please
amor
executor!
Z--
can
probably
do
a
better
job
doing
it.
So
this
is
what
diamond
allocation
does
and
it
is
getting
added
to
does
kubernetes
native
support.
Gpu
support
is
you
know,
able
to
propagate
information
to
kubernetes
for
GPUs.
B
You
know,
and
what's
almost
interesting
for
for
us
as
well-
is
that
we
have
a
lot
of
large-scale
use
cases
from
Hadoop.
What
you
need
is
also
a
much
better
batch
scheduling
when
it
comes
to
a
lot
at
certain
stage.
Right,
you
don't
want.
You
want
all
your
teams
and
organizations
able
to
have
a
fair
shares
of
your
workloads
and
resources.
B
So
there's
six
scheduling,
there's
cube
batch,
that's
happening
right,
and
definitely
that
is
the
right
direction,
but
I
think
most
large-scale
use
case
and
enterprises
will
need
something
like
this
you'll
be
able
to
actually
support
resource
management
at
scale
for
your
batch
jobs,
all
right
so
really
quickly.
What's
next
right
talk
about
spark
own
kubernetes,
just
one
big
one
piece
of
getting
your
cloud
native
machine
learning
working
on
the
kubernetes
coming
from
the
hyper
pilot
world.
B
This
is
also
why
I
really
firmly
believe
is
that
at
scale
things
are
harder
to
write
your
tap
to
tune
or
container
resources.
You
have
to
tune
your
spark
resources,
the
other
tune,
your
VM
cluster
size
instance
types
all
kinds
of
stuff.
So
we
really
believe
that-
and
this
is
just
machine
learning
right
if
user
actually
starts
to
actually
cram
in
data
warehousing
data
engineering
right
in
this
shared
environments,
how
do
you
it's
almost
impossible
for
it
engineers
an
admin
to
really
tune
everything
that
has
your
best
case,
utilization,
right
and
I?
B
A
B
Screenshots,
but
this
is
what
we're
also
looking
to.
Is
she
in
bed
into
calderas
future
offerings
in
some
points,
but
using
data
you
can
really
able
to
figure
out
a
lot
more
insights
right
rather
than
just
telling,
rather
just
telling
your
data
engineers
hey.
Your
container
is
CP
really
just
seeing
CPU
spiked
at
80%,
and
rather
just
seeing
that
hey
my
memory
has
been
using
a
60%
I
got
wound
killed,
you
know.
Can
we
actually
tell
you
what
really
happens?
B
You
know
what
what
is
a
key
resource
bottleneck,
because
if
you
haven't
you
look
at
your
monitoring
dashboard,
it's
really
hard
to
figure
out
if
networking
as
high
CPU
as
high
as
well
right.
If
it's
IO
is
high
in
the
cloud
environments,
you
actually,
your
CPU
might
be
high
too
right,
because
you're
have
a
shared
volume.
Actually
iowa's
high
networking
might
be
high
as
well.
B
You
have
network
volume
is
attached
right,
really
hard
to
figure
out
what
is
a
key
resource
bottlenecks
and
what
the
key
problems
are
happening
and
is
it
interference
miami
getting
actually
overtaken
by
another
container
running
on
the
same
node
or
cluster?
So
we
did
a
lot
of
work
to
do
some
machine
learning
to
actually
figure
out
what
insights
and
figure
out
the
root
causes.
So
this
is
one
screenshot
there
I
want
to
quickly
mention
lasting,
is
just
showing
you
what
to
be
done.
B
What
is
the
key
problem
doesn't
really
automatically
solve
and
give
you
the
best
utilization
right
to
get
the
best
utilization
possible?
You
really
need
to
have
automation,
and
what
we
talked
about
automation
is
not
just
adding
more
nose.
It's
not
just
adding
more
pots.
You
know
in
this
particular
example.
If
you,
if
you're
interested,
you
definitely
should
look
for
search
for
hyper
pilots,
and
you
can
definitely
see
a
lot
more
details.
B
We
we
seeing
a
lot
of
use
cases
where
there's
interference
happening
on
the
same
note,
if
I
have
a
pod
running
a
lot
of
network
traffic
and
overtaking
another
long-running
service
and
in
a
machine
learning
context,
this
is
a
model
deployments.
That's
running
next
to
your
spark
shop,
doing
batch
job
training.
So
how
do
I,
if
you
don't
want
to
waste
a
lot
of
resources
and
pre,
allocates
your
largest
cluster
to
do
model
deployments?
B
You
want
to
you
to
a
bolita
lies
your
slack
resources
to
do
model
training,
but
it's
a
real
challenge
to
not
interfere
with
the
deployments
that
are
doing
inference
on
real
time.
So
in
this
screen
it
is
probably
a
little
hard
to
digest,
but
this
is
saying
that
this
is
what
long-running
service
is.
What
happens
when
you
run
a
bath
shop
next
to
your
long-running
services?
It's
basically
a
latency
it
spikes
up,
because
this
is
running
sparks
jobs
or
doing
you
know,
Tara
sorts
and
doing
a
lot
of
different.
B
B
The
whole
runtime
is
running
but
able
to
actually
figure
out
what
is
a
key
resource
bottlenecks,
we're
talking
about
IO
level,
network
level,
CPU
memory
and
even
lower
levels,
things
and
automates
it,
because
you,
if
we
understand
your
bass
job,
it
doesn't
have
to
finish
right.
The
next
minutes,
you
don't
really
need
to
take
all
the
resources
you're
looking
for
and
only
give
you
certain
resources
that
required
acted
at
real
time.
Then
I
can
able
to
really
achieve
much
higher
utilization,
so
your
utilization
can
jump
much
higher
without
really
killing
your
job
spatula.
B
So
that's
the
next
kind
of
level
of
automation,
ruling
on
a
label
for
communities.
So
that's
it's
all
the
stuff.
We
talked
about
life
spark
on
Calder
and
machine
learning,
you're
gonna
feel
free
to
come
and
sign
up.
We
have
a
private
preview
coming
up,
good,
can
go
to
URL
and
actually
try
out
and
give
us
feedback
early.
This
runs
on
all
kubernetes
manage
kubernetes
and
yeah.
There's
also
relevant
six.
That
would
just
talk
about
scheduling
in
Big
Data.