►
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Kubeflow From the End User’s Perspective: The Good, The Bad, and The Ugly - Xin Zhang, Caicloud
Kubernetes has become the most popular open-source container orchestration platform for managing cloud-native workloads. Furthermore, given the ubiquitousness of Artificial Intelligence (AI), Kubeflow has risen as a new open-source project tailoring Kubernetes to optimize for Machine Learning (ML) stacks, solving the devops chores and performance bottleneck usually plaguing a production ML system. Despite the promise, we (as a contributor to Kubeflow) have identified glitches of Kubeflow when being applied in the wild. We have surveyed 50+ real Kubeflow users (both system administrators and machine learning developers) from our enterprise customers. We will share how Kubeflow solves their pain-points, what pitfalls and disappointments they encountered in daily usage scenario, and how to evolve Kubeflow to be more practical and generally applicable..
To learn more click here: https://sched.co/FuJx
A
B
A
B
A
B
B
B
A
A
A
B
Think
that
we
call
marginalize
AI
that
they
thought
that
the
most
important
thing
is
about
the
algorithms
is
about
the
ML
code.
So
this
is
the
core
business,
of
course,
success
factors
for
the
business
and
the
other
really
engineering
related.
They
thought
it
was
contributor,
not
so
relevant.
However,
the
leaders
of
Google
cloud
called
Angela
Moore.
B
B
B
B
B
Mode,
so
you
need
to
out
aggregate
data
second
barrier.
Your
second
challenge
is
Albert
algorithmic
differentiators.
You
need
to
have
very
good
one-of-a-kind
algorithms
if
you
stay
one-of-a-kind
second.
Lastly,
if
the
product
do
you
have
a
bottom
up
product
with
your
own
data,
your
own
capacity
to
have
a
very
perfect
product
that
is
empty
to
end,
so
open
source
is
very
popular
and
the
algorithms
challenge
can
be
solved
in
recent
years.
A
A
B
Are
all
the
product
line?
It
requires
a
lot
of
jobs
so
who
develop
those
models?
They're,
not
they
don't
have
any
background
about
machine
learning
PhD.
They
are
software
engineer,
so
I
asked
them.
Do
you
know
math?
Do
you
know
statistics?
They
said
we
don't
have
to
because
we
have
tools.
We
have
open
source
in
neural
networks,
and
all
we
want
to
do
is
to
use
these
tools
to
use
cloud
computing
to
fast
quickly,
develop
a
20
model
so
to
be
more
specific
when
it
comes
to
cube
flow
or
AI
pipelines.
B
If
we
split
it
into
more
detailed
steps,
a
lot
of
the
stabs
are
mathematic
related
that
are
related,
and
then
we
need
to
have
a
very
solid
foundation
with
data
and
there
was
a
training
and
after
the
training
model
was
launched,
and
then
we
need
to
manage
it.
Just
like
managing
a
java
application,
you
need
to
manage
your
model
after
it
was
launched.
So
this
is
a
very
detailed
pipeline
for
AI
commercial
foundations.
First,
you
need
to
have
a
mathematic
algorithms
and
then
you
need
to
so.
B
B
Been
in
this
game
since
the
beginning,
another
big
contributor
for
us
is
we
come
up
with
the
cube
flow
logo,
so
we
have
the
cube
flow
local
designer
here,
which
is
it.
She
is
the
cube
slope
local
designer.
You
can
see
that
this
is
a
very
beautiful
local
contra
by
mist
route
until
evening,
and
also
Dave
is
also
here.
As
you
can
see,
this
is
a
code.
The
credit
is
go
post
to
David
our
check/uncheck.
B
This
is
the
cube
flow
pipeline
and
this
is
the
English
version
of
the
video
to
flow
some
of
the
some
steps.
So
we
have
this
big
picture
and
then,
let's
look
at
the
status
quo
of
the
cube
flow.
What
kind
of
questions
is
solved?
What
kind
of
problem
is
solved
now?
Our
key
flow
is
transfer
from
point
32.4
version,
so
it
can
actually
solve
some
problems
in
some
key
steps.
For
example,.
A
B
When
it
comes
to
developing,
to
use
mostly
notebook
using
this
approach,
additionally
model
online
in
real
time
and
then
when
it
comes
to
kind
of
training.
So
how
can
we
make
sure
that
we
can
have
large-scale
training?
We
use
kubernetes
dispatch
capacity
to
define
the
resource.
We
can
run
it
perfectly
on
terminated
platforms
and
when
it
comes
to
serving,
who
use
tensor,
slower,
tensorflow,
serving
serving
modules
and
also
is
selling
community
related
tools.
B
So
I
just
help
you
to
review
the
case
law
I
believe
that
Dave
will
continue
my
presentation
to
be
more
detailed
about
Q
flow.
What
is
it,
why
is
it
so
now,
let's
go
right
into
our
main
topic
today,
we
want
to
talk
about
the
cute
flow
since
its
beginnings,
our
practice
with
cute
flow.
What
kind
of
experience
what
kind
of
lessons
we
learned
from
using
cute
flow
in
the
past?
If.
B
Their
requirements,
their
behavior
and
their
preference.
This
way
we
can
feed
back
to
our
own
team
service
better.
So
all
of
my
education
was
based
on
the
report
that
we
published
and
we
published
that
report
on
the
local
community,
and
actually
you
can
found
that
paper.
It
is
more
detailed
about
the
questions
and
problems
we
found
out.
So
if
you're
interested,
please
click
the
link,
you
will
find
our
paper.
So
let
me
tell
you
about
some
of
our
findings.
First,
to.
B
Want
to
do
this
product,
we
need
to
make
sure
who
is
our
target
customer
who
is
our
end
user?
So
in
this
context
there
are
different
stakeholders.
There
are
business
analysts,
there
are
staffer,
there
are
data
scientists,
we
create
the
model
and
there
are
also
data
engineers
in
charge
of
the
overall.
A
is
the
use
of
structure
architect
and
also
we
have
IT
and
devil
guys.
So
there
are
three
little
important
titles
here
on
this
lies.
B
We
think
that
the
business
analyst
is
not
the
most
important
stuff
I
believe
that
cubes
law
need
to
really
empower
developer,
to
make
all
the
developers
to
be
to
train
their
models
in
the
more
efficient
way.
So
previously,
I
talked
about
the
end
user,
who
is
our
end
user
of
cubes
law
and
now
I'm
thinking
I
like
to
introduce
three
different
end
users
for
our
customer
project,
because
isn't
a
customer
have
different
portrait?
Okay,
so
those
are
all
using
our
cube
flow
and
let
me
introduce
their
customers
portrait
so
I.
A
B
First
is
the
internet
Giants
for
IT,
Giants
or
IT
companies.
They
use
a
lot
of
very
advanced
IT
platform.
Second,
there
are
traditional
industry,
color
players
like
energy
state
grid
like
finance,
even
manufacturers.
They
are
traditionally
players
in
this
industry
and
the
third
customer
is
more
advanced
even
the
first
categories.
There
are
the
research
institutes
like
universities,
like
research
centers
in
the
company,
so
we
found
that
these
three
customers
for
us
when
they
were
using
AI
platforms,
they
have
different
that
they
have
a
similar,
would
have
different
departments.
B
A
B
Very
advanced
technologies,
so
these
are
some
of
our
findings
about
three
different
kinds
of
customers,
and
also
there
are
some
empty,
encourages
or
counterintuitive
findings.
We
found
that
in
the
past
we
thought
that
most
a
lot
of
companies
who
use
a
lot
of
CPU
utilization
requires
a
I
had
a
cube
floor.
However,
in
reality
we
found
that
a
lot
of
customers-
they
don't
have
a
GPU,
what
they
don't
have
so
much
gpo
utilizations.
They
just
use
a
very
typical
word,
typical
AI
machine
learnings.
B
They
won't
just
to
want
to
use
machine
learning
to
help
them
their
business
for
those
companies
they
have
of
strong
requirements
to
use
AI
platform.
So
that
is
the
biggest
pain
point.
That
is
the
biggest
questions
acutely
oSIsoft
not
only
allocated
resource
not
only
manage
the
GPU.
We
found
that
there
is
a
lot
of
customers.
They
can
finish
their
training
jobs
on
one
servers
with
8
GPU.
A
A
C
C
C
C
A
C
A
A
C
A
A
A
A
C
A
C
C
A
C
B
C
C
A
C
A
B
A
C
C
A
A
A
A
A
B
A
A
A
A
B
C
C
C
B
C
C
A
A
B
C
A
B
On
the
other
hand,
according
to
LinkedIn,
the
posting
for
data
scientists
over
past
five
years,
job
post
for
related
related
to
data
scientists
increase
by
tenfold
over
the
past
five
years.
So
we
need
these
data
scientists
that
will
help
us
to
pinpoint
the
question
but
people
the
problem,
but
it
is
very
hard
to
hire
them.
So
how
do
we
solve
this?
There's
a
cop's
for
solutions.
We
have
another
paper
from
Google.
B
They
provide
a
online
service
with
a
black
box
upgrading
it's
a
black
past
optimizations
to
help
you
to
fine-tune
your
parameters,
examples,
ml
and
in
cube
flow.
There's
a
subgroups
called
Katie.
So
using
a
similar
ideas,
follow
the
same
line
of
thinking
and
to
combine
it
with
model
DB
to
do
multi,
experiment,
tracking
and
comparison.
B
And
also,
we
have
other
solutions
to
solve
that,
for
example,
if
it
is
a
new
context
rather
start
from
scratch,
we
can
actually
transfer
our
learning
if
we
could
actually
copy
other
models.
For
example,
we
have
TF
hub,
and
in
recent
years
we
have
AI
hub
just
the
unity
private
alpha
state,
so
with
TF
hub.
We
can
also
help
you,
alongside
with
learning
transfer.
We
can
accelerate
the
fine
tuning
of
the
model.
B
Let
me
speed
it
up
to
leave
some
time,
for
it
don't
mr.
tone
for
a
Q&A
sessions.
This
is
our
solution,
which
is
very
easy.
Simple
help
you
to
improve
the
brain,
fine
grain
of
the
GPU.
How
can
we
have
a
better
management
and
marketing
of
the
GPU?
How
can
we
have
distributed
training
for
different
systems
and
during
the
host?
How
can
we
just
like
Java
Script?
How
can
we
do
the
a/b
test
better
and
do
the
better
CI
CD?
So
in
the
pipeline?
We
also
have
similar
ideas.
A
B
A
better
iteration
is
better
inference
for
any
user.
So,
with
this
model
training,
this
is
a
new
system.
For
us
we
called
cloud
native
intelligence
based
on
cloud
native.
We
can
empower
the
intelligent
application
systems.
What
kind
of
thing
we
can
do?
Well,
let
me
give
you
an
example:
there
are
some
end-user
applications,
for
example,
on
the
e-commerce
platforms.
B
If
we
want
to
launch
ourselves
in
the
new
market
in
overseas,
we
can
actually
use
this
model
to
test
and
also
we
help
the
fashion
to
classify
their
pictures
and
mark
their
pictures,
and
although
we
can
provide
service
to
OCR
to
government
and
to
help
them
empower
the
developers
to
do
more,
last-minute,
any
questions,
any
questions
you
can
raise
questions
to
mr.
Tom
mr.
Tom.
B
B
We
found
that
there's
a
resource
scarcity
when
we
are
doing
training.
So
how
can
we
estimate
how
much
resource
the
model
needs
during
training
is
a
very
tricky
question?
Okay,
so
I'm
not
involved
in
this
talk,
I,
don't
know
why
they
add
my
name
on
this
chart.
But
let
me
answer
your
question,
so
it
is
somehow
they're
hard
to
estimate
resource
because
we
need
we
ask
all
the
jobs
to
have
limited
resources
ations.
B
Up
to
now,
there
are
several
solutions
to
estimate
the
resource,
how
much
resource
we
need.
Well,
first,
we
have
all
those
resource
offline,
we're
separated
so
because
kubernetes
provide
reserved
resources.
So
we
try
to
make
sure
that
communities
and
dark
agent
should
not
fail.
That
is
the
micro
service.
We
have
some
fun
things,
for
example,.
A
B
They
are
doing
their
own
business
when
the
companies
are
doing
their
own
business,
all
of
their
part,
all
of
their
container,
all
of
their
jobs
tasks.
And
what
is
the
department
requires?
They
use
annotation
to
tell
our
platform
and
then,
when
we
are
doing
this
patch,
there
are
some
extensions
where
we
can
do
and
our
control
well
found
their
historical
data
to
suggest
some
research
to
them
and
suggest
some
data,
so
we
will
make
it
transparent
to
our
end
users.
The
end
users
can
look
at
our
suggestions
and
tend
to
allocate
a
resource.
B
We
don't
do
automations
because
it
is
very
risky.
We
just
do
suggestions
so
based
on
a
texta.
We
based
on
their
data
usage,
their
models,
whether
they
are
using
picture
classifications.
What
is
the
size
of
their
model
so
based
on
all
of
those
data?
We
will
make
suggestions
for
you.
So
then,
according
to
our
suggestion,
they
will
have
request
and
limit.
So
these
will
live
nowhere
to
tear
questions
and
also
we
will
have
some
buffers
at
at
the
end,
and
previously
I
talked
to
Ally's
colleagues.
B
B
Some
of
them
are
self-made,
so
when
it
comes
to
keep
flow,
it
was
consist
of
different
components,
providing
a
lot
of
components
that
can
be
plugged
in,
for
example,
there
are
10
to
20
components
was
wrong
and
famously
we
talk
about
pipelines
to
integrate
all
different
into
components.
So,
in
our
platform
there
are
two
or
three
components
was
used
by
us.
They
are
independent
authorities
perished
because
within
the
ecosystems
reading
the
AI
ecosystems,
the
machine
learning
requires
a
different
tools.
We
cannot
integrate
all
the
tools.
We
just
choose
to
three
components:.
B
A
B
As
we
have
two
questions,
the
first
one
in
your
slides,
you
mentioned
packet
on
so
actually
I
used
to
research.
In
that
there's
a
support.
There
is
a
PF
as
a
support
to
data
set.
I,
don't
know
what
is
the
future
for
this.
Please
tell
us
in
a
second
towards
the
end
of
the
presentation.
You've
said
that
you
have
optimization
to
CPU
and
GPU,
so
we
don't
find
it
on
cube
flow
website.
So
do
you
want
to
open
that
in
the
future,
or
this
is
just
a
private
project
for
you?
Thank
you.
B
So
the
question
I
can't
forget
your
first
question.
Let
me
answer
the
second
one.
So
when
it
comes
to
GPU
optimization
when
it
comes
to
cube
flow,
there
is
no
optimization
for
GPU
David,
probably
but
talk
about
this,
and
you
can
ask
him
for
us
when
it
comes
to
gpo
optimization.
We
already
do.
If
you
want
to
share,
we
can
change
the
kubernetes
device
plug-in
using
30
TPU,
so
you
can
request
you
said
well,
you
could
probably
say
that
we
we
can
provide
an
independent
GPU
for
you.
B
But
if
you
want
to
share
this
with
others,
then
we
can
provide
you
with
three
to
five
GPU.
You
can
share
this
with
others.
You
can
share
the
capacity
with
others.
So
this
is
what
we
were
doing
based
on
cuban
ethics.
We
will
not
change
cuba
and
the
first
question
is
about
GFS,
so
we
didn't
use
that
we
focus
their
own
data
management
ecosystem
personally
speaking
because
they
didn't
provide
so
much
notification.