►
From YouTube: Taming the Beast: Managing the day 2 operational complexity of Kubeflow - Mofi Rahman & Paul van Eck
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon North America 2021 in Los Angeles, CA from October 12-15. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Taming the Beast: Managing the day 2 operational complexity of Kubeflow - Mofi Rahman & Paul van Eck, IBM
Kubeflow is a machine learning toolkit for Kubernetes where users can develop, deploy, and manage ML workflows in a scalable and portable manner. Kubeflow is composed of many components such as notebooks, service meshes, and pipelines, coupled with their many potential configurations and dependencies. All these moving parts can be daunting for an end user and make Kubeflow impossible to manage and maintain even for an experienced teams. To help alleviate these woes, this session will provide users some insight into how they can navigate this confusing landscape. Tips regarding deployment and configuration, as well as lessons learned from both managing Kubeflow deployments and contributing to upstream Kubeflow, will be provided to help users tame the platform’s operational complexity.
B
A
So
welcome
to
our
talk
about
taming,
the
beast
and
managing
the
day
two
operational
complexity
of
kubeflow.
Here
we
hope
to
share
some
insights
about
how
to
make
kubeflow
operations
a
bit
less
daunting.
So
let's
get
started
yeah.
So
if
you
are
here
for
kubernetes
ai
day,
then
you
are
probably
already
familiar
with
or
at
least
know
about,
google
flow.
A
So
taking
a
look
at
a
conceptual
overview
diagram.
We
can
see
kubeflow
as
a
platform
for
arranging
the
components
of
your
ml
system
on
top
of
kubernetes
and
as
you
can
see,
there
are
quite
a
number
of
components
that
are
typically
involved
in
a
kubeflow
deployment,
and
this
is
by
no
means
all-inclusive.
A
Now,
with
all
these
bits
and
pieces,
you're
bound
you're
bound
to
run
into
some
trouble
right.
Well,
that's
the
nature
of
the
beast.
When
you
are
dealing
with
so
many
moving
parts,
it's
no
question
that
deployment
can
be
challenging.
You
might
run
into
integration
issues
when
trying
to
install
kubeflow
the
kubeflow
stack
with
existing
components
in
your
cluster.
A
Maybe
the
install
of
your
kubeflow
distribution
requires
additional
configurations
on
top
of,
what's
already
there
and
for
those
who
have
looked
in
the
kubeflow
manifest
repository
in
the
past,
you
may
have
found
the
sea
of
yaml
files
and
folders
quite
confusing,
and,
furthermore,
managing
kubeflow
can
also
be
somewhat
tricky
when
doing
day.
2
operations
such
as
updating
or
reconfiguring
a
component
so
like
most
large
pieces
of
software
kubeflow,
has
its
own
set
of
obstacles
and
complexities.
A
A
Now,
first,
let's
talk
a
bit
about
deployment
when
you
deploy
a
kubernetes
app.
This
requires
applying
a
set
of
yaml
files
into
your
cluster
in
order
to
create
the
necessary
the
necessary
resources
for
kubeflow.
All
the
application
resources
can
be
found
in
the
kubeflow
manifest
repository
now
generally,
when
people
deploy
kubeflow
people
typically
use
the
command
line
tool.
Kf
cuddle.
This
was
created
to
deploy
something
called
a
kf
def
file,
which
is
a
yaml
file
that
contains
references
to
well
a
list
of
references,
two
of
applications
you
want
to
deploy
for
kubeflow.
A
So
this
was
this
kfc
and
kfdf
paradigm
was
heavily
used
in
prior
releases,
with
many
distribution
saw
guys
using
install
guides
using
this
tool.
However,
recently
there
has
been
a
push
by
the
community
to
make
kubeflow
installable
using
just
customize
and
coupe
cuddle,
which
are
both
tools
used
commonly
in
the
wider
kubernetes
ecosystem,
so
that
is
also
an
option
and
will
likely
be
the
favorite
option
going
forward.
A
A
A
First,
you
have
the
apps
folder,
which
contains
kubeflow's
official
components
as
maintained
by
the
respective
kubeflow
working
groups.
Then
we
have
the
common
folder,
which
contains
kubeflow
common
services
as
maintained
by
the
manifest
working
group.
These
include
services
such
as
istio
k,
native
and
even
cert
manager.
Then
there
is
a
contrib
folder
which
contains
third-party
contributed
applications.
A
These
are
maintained
externally
and
don't
are,
and
not
part
of
a
kubeflow
working
group,
at
least
not
yet,
then,
there's
also
this
distributions
folder,
which
contains
manifest
for
specific
opinionated
distributions
of
kubeflow.
But
the
plan
is
to
phase
this
folder
out
and
going
forward.
New
distributions
of
kubeflow
will
be
developed
outside
of
the
kubeflow
github
org,
and
manifest
repository
to
also
help
clean
up
and
consolidate
manifest
development.
A
A
A
A
There
is
improved
accountability
for
maintaining
component
manifest
in
general
component
developer
component
developers
will
have
an
easier
time
of
maintaining
their
manifest,
for
example,
in
kf,
serving
all
manifests
are
located
in
the
config
directory
in
the
project
repository
the
default
install
is
for
the
standalone
experience
with
kf
serving
how,
however,
to
support
integration
with
the
rest
of
the
kubeflow
platform.
Kf
serving
provides
a
kubeflow
overlay,
from
which
a
user
can
do
a
customized,
build
piped
to
a
coupe
color
apply
to
install
a
kubefloor
kubeflow
compatible
kubeflow
platform
compatible
version
of
the
component.
A
This
leads
into
the
next
few
points
where
the
hope
is
to
have
a
smoother
deployment
experience
and
increased
modularity
kubernetes
users
again
are
likely
already
familiar
with
customizing
coop
cuddle.
So,
with
these
changes,
one
can
more
easily
install
individual
components
based
off
of
their
needs.
A
Now
it
should
also.
It
should
also
be
more
clear
how
the
common
manifest
for
services
like
k-native
and
sto
differ
from
what
is
provided
upstream.
For
example,
the
manifest
for
the
kubeflow
1.3
release
include
is
to
1.9
in
the
sa
1.9
folder.
It's
documented.
What
changes
were
made
to
get
is
to
integrated,
with
the
other
kubeflow
components,
making
it
easier,
making
it
easier
for
users
to
integrate
their
existing
deployments
of
these
services.
A
A
Kubernetes
can
handle
stateless
applications
just
fine
by
itself.
However,
when
you
start
adding
stateful
components
like
a
database
or
monitoring
system,
kubernetes
does
not
know
how
to
scale
upgrade
and
reconfigure
these
types
of
things.
These
require
domain,
specific
knowledge
or
human
operational
knowledge.
A
So
the
idea
behind
operators
is
to
replace
what
would
be
what
would
be
a
human
operator
managing
stateful
applications
with
a
software
operator
which
would
have
the
knowledge
to
be
able
to
do
all
the
functions
that
typically
a
devops
team
would
be
doing,
though,
through
the
use
of
a
custom
control,
loop
and
crds
custom
resource
definitions.
Along
with
the
needed
domain
specific
knowledge,
a
well-made
operator
is
able
to
implement
and
automate
common
day,
1
and
day
2
operations.
A
A
Currently,
the
kubeflow
operator
relies
on
kf
cuddle
and
is
built
on
top
of
a
kfdf
custom
resource,
though
this
may
possibly
change
in
the
future.
Considering
all
the
manifest
changes,
in
any
case,
depending
on
your
cloud
environment,
deploying
kubeflow
to
your
cluster,
can
be
as
easy
as
a
few
simple
steps,
since
it's
available
on
operator
hub,
a
catalog
of
community,
published
operators
and
one
benefit
of
using
the
kubeflow
operator
is
that
it
will
help
monitor
all
the
child
resources
of
your
kubeflow
deployment,
which
can
be
a
lot.
A
A
B
B
The
security
patches
troubleshooting
of
when
something
does
go
wrong,
and,
finally,
monitoring
to
know
when
how
things
are
working,
how
resources
are
being
utilized,
one
of
the
biggest
challenges
of
queue
flow
being
the
open
source
project
that
relies
on
a
lot
of
other
open
source
project
is
that
qfloor
has
a
lot
of
moving
parts.
Each
component
has
their
own
release
cycle.
Each
component
has
their
own
update
and
upgrade
path.
So
the
way
you
can
upgrade
one
component
is
not
the
same
as
upgrading
another
component.
B
It
is
all
built
on
top
of
open
source
technology,
which
is
also
changing
very
rapidly.
Qflo
is
deployed
on
kubernetes,
where
kubernetes
itself
is
coming
from
different
vendors
and
platform.
Each
platform
has
very
minor
differences
that
eventually
add
up
to
be
pretty
big
changes
when
you're
talking
about
your
keyflow
deployment.
This
is
by
no
means
a
complete
list
of
things
you
can
do
to
solve
all
your
q
flow
problems,
but
I
think
we
can
help
you
get
started
with
showing
you
some
of
the
common
issues
and
how
we
had
solved
them
in
the
past.
B
This
should
give
you
some
idea
what
you
can
expect
to
do
when
you
bump
into
some
of
these
problems
yourself.
One
of
the
common
cases
we
see
is
that
component
has
an
updated
version
where
keyflow
release
hasn't
caught
up
to
that
version.
Yet
so
the
user
wants
to
use
the
latest
version
of
some
component,
where
we
don't
have
that
on
the
keyflow
manifest
yet,
and
we
can,
we
might
want
to
do
that
for
several
reasons.
One
of
them
being.
B
B
B
B
If
you
are
seeing
issues
around
the
failed
calling
web
books,
it's
because
you
might
have
some
scale
web
hooks
hanging
about,
so
you
can
delete
them
using
the
command
in
the
slide
below
oftentimes,
when
you're
deploying
keyflow
you'd
access
your
keyflow
dashboard
by
exposing
the
node
port
on
your
cluster's
public
ip
of
your
cluster
node.
But
that
is
not
a
secure
solution.
So
if
you
want
your
application
to
be
used,
if
you
want
to
use
production
data,
you
want
to
have
added
security
by
enabling
https.
B
If
you
are
able
to
update
the
service
type
of
eastern
ingress
gateway
to
load
balancer,
you
can
use
cert
associated
with
domain
name
of
external
ip.
Some
cloud
providers
may
already
provide
tls
certificate
or
you
can.
You
are
able
to
generate
one
for
your
service
and,
if,
once,
if
you
can
do
that,
you
can
create
the
certificate
into
istio
system
namespace
as
a
secret
and
then
mount
that
secret
into
your
service
and
then
update
the
queue
flow
gateway
and
the
change
port
to
https
and
port
443
and
add
the
tls
section.
B
So
when
you
are
trying
to
secure
your
k
native
deployment
for
kf
serving,
you
may
need
to
take
a
couple
of
extra
steps
for
one.
You
will
need
a
valid
custom
domain
for
canadian
app
routing.
You
may
need
to
update
the
k
native
domain
format.
You
also
may
need
to
add
wildcard
matching,
because
k
native
would
add
some
zone
on
top
of
your
domain
name
for
on
the
serving,
so
you
might
need
to
add
a
wildcard
matching
on
your
host
name
on
your
q
flow
gateway.
B
If
you
are,
if
you
were
using
an
older
version
of
qflow
you,
you
probably
had
to
do
some
workarounds
to
make
sure
upper
higher
version
like
st
1.6.
We're
working
with
q
flow,
I'm
not
going
to
go
into
details
of
that
process
anymore,
because,
hopefully,
with
the
newer
versions
of
keyflow
that
comes
out,
this
problem
will
be
non-existent
for
update
and
upgrade.
There
are
a
few
different
things
to
think
about
again.
Number
one
is
keyflow
component
update
as
queue
flow.
B
New
tools
come
out,
keyflow
is
underlying
component
of
keyflow
can
update,
and
you
will
have
to
be
mindful
and
think
about
how
you
want
to
update
those
components.
You
also
need
to
think
about
kubernetes
update
we're
going
to
talk
about
a
little
bit
about
that
in
later.
I
also
need
to
think
about
bug
fixes,
as
as
security
patches
come
out.
What
fixes
come
out
for
different
components?
How
do
you
want
to
go
about
updating
those
components
and
finally
platform-specific
changes?
B
You
might
be
managing
your
kubernetes
distribution
yourself
or
you
might
be
using
a
managed,
kubernetes
distribution
on
any
of
the
major
cloud
providers.
So,
if
you're
using
a
managed
kubernetes
each
of
the
platform
can
have
minor
differences
in
terms
of
their
storage
classes
in
terms
of
their
networks
and
you'll
have
to
be
mindful
of
how
those
differences
mean
you
will
have
to
have
a
different
queue
flow
installation.
B
Another
thing
to
be
mindful
of
that.
If
you
want
to
persist
your
data,
one
good
way
to
do,
that
is
using
external
databases,
qfl
installation.
You
can
install
a
database
as
part
of
the
keyframe
installation
itself,
but
as
a
as,
if
you're
running
this
database
inside
the
kubernetes
cluster,
as
you
have
to
do
the
upgrade
you
will
have
to
a
keep
on
managing
the
database
and
b.
It
will
be
harder
to
update
and
upgrade
the
database
without
losing
the
data.
B
Keyflow
runs
on
top
of
kubernetes,
so
you
have
to
kind
of
adhere
to
the
kubernetes
update
policies
as
well.
So
kubernetes
is
on
an
around
three-month
release
cycle
and
current
version
of
kubernetes
gets
deprecated
in
about
nine
months.
So,
if
you
are
on
kubernetes
version,
120
now
by
the
end
of
2021,
kubernetes
version,
123
or
122
will
come
out.
An
older
version
will
start
getting.
Deprecated
deprecations
in
kubernetes
are
announced
about
three
four
versions
in
advance.
B
B
So
we
talked
about
some
some
general
practices
you
could
use
to
help
mitigate
the
problems.
You
see
some
common
issues
we
have
seen
before
and
how
you
can
start
thinking
about
it.
But
if
you
hit
a
roadblock
in
your
hub
flow
distribution,
how
do
you
go
about
finding
information?
How
do
you
go
about
finding
help?
Keyflow
has
a
public
slack
channel
that
you
can
ask
questions,
connect
to
people
that
are
used
facing
similar
issues
or
have
solved
the
same
issues
that
you
are
facing?
B
You
can
open
an
issue
on
the
github
repository
keyflow
team
tries
to
help
folks
as
much
as
possible
and
again
finding
issues
also
helps
us
improve
the
project
as
we
can
fix
the
bugs
as
they
come
around,
but
there
is
still
a
question
about
how
do
we
debug
failure
and
that's
a
question?
It's
very
difficult
to
answer,
because
each
of
those
problems
can
come
from
a
number
of
different
places,
so
some
common
ways
to
figure
out
exactly
what
the
issue
is
and
how
can
we
solve
them
is
first
identify
the
source
of
failure.
B
So
is
it
a
failure
from
a
key
flow
component?
Is
it
a
failure
from
istio
or
is
it
a
problem
from
kubernetes?
One
common
way
I
like
to
deal
with
these
kind
of
problems
is
look
for
similar
issues.
If
the
issue
has
been
earned
for
a
while
and
hasn't
had
any
resolution,
this
might
not
be
the
fastest
way
to
get
a
solution,
but
sometimes
it
can
nudge
you
towards
the
right
direction.
Look
at
the
logs
from
the
failing
pod.
B
Sometimes
the
log
itself
will
tell
you
exactly
why
it's
failing
some
of
the
common
log
errors
you
can
see.
Is
that
permission
denied
or
in
the
file
found
or
like
some
network
connection
timed
out,
so
this
can
kind
of
guide
you
towards
the
right
direction
of
where
the
error
is
coming
from
in
the
queue
flow
slack
you
can
ask
folks,
there
are
always
people
online
and
if
someone
else
has
seen
the
same
problem,
they
can
definitely
help
you
out
and
help
solve
your
problem
and
finally
raise
an
issue
on
the
repo.
B
This
doesn't
have
to
be
your
last
step
if
you
are
facing
problem-
and
you
think
the
problem
is
coming
from.
The
key
flow
project
on
the
software
itself
definitely
open
an
issue
so
that
the
maintainers
can
take
a
look,
and
these
definitely
help
us
improve
the
quality
of
the
project
over
time.
Finally
trouble.
How
do
you
go
about
troubleshooting
problems
that
are
specific
to
individual
platforms?
B
Some
issues
can
be
platform.
Specific
each
platform
has
minor
differences
that
can
lead
to
situations
that
are
not
covered
by
general
q
flow
for
that
definitely
use
provider.
Specific
docs
use
github
issues
to
mention
exactly
which
provider
kubernetes
you're
using
and
what
are
the
issues
you're
saying
and
in
the
public
slack
there
are
channels
specific
to
individual
vendors.
So
definitely
there
is
your
issue
there,
as
well.
So
maintainers
from
those
companies
and
vendors
can
look
at
that
issue
and
help
you
out.
If
they
can
monitoring.
B
So
monitoring
has
couple
of
aspects.
Number
one
is
figuring
out
how
much
resource
you
are
using
in
your
cluster
and
figuring
out
if
your
components
are
healthy
and
for
this
keyflow
doesn't
necessarily
have
any
specific
opinions
on
how
you
go
about
doing
that.
Keyflow
is
using
hd
underneath
and
he
still
has
a
lot
of
monitoring
capabilities
by
using
things
like
prometheus
grafana
alert
manager.
B
So
you
can
use
all
of
those
tools
at
your
disposal
to
get
information
from
your
cluster
to
know
exactly
how
much
resource
is
being
used
and
how
your
components
are
behaving
keyflow
1.3
release
has
been
around
for
a
while.
With
this
release
comes
many
new
upgrades
to
keyflow
that
will
help
resolve
some
of
the
issues
we
have
been
seeing
in
the
past,
for
example,
with
the
update
to
istio
1.9.
B
Finally,
the
manifest
repository
is
being
reorganized,
as
paul
mentioned,
so
we're
going
to
have
much
easier
installation
of
keyflow
and
it
will
help
it
will
be
easier
to
understand
the
manifest
repository
on
how
keyflow
is
working
and
the
official
installation
will
rely
on
cube,
cuddle
and
customize.
The
kfdf
and
kf
cuddle
will
still
be
around.
It
will
be
still
be
possible
to
install
q
flow
using
kf
cuddle,
but
the
official
installation
instruction
will
be
focusing
on
customize
and
keep
cuddle.
B
Some
of
the
reference
links
are
here
as
the
manifest
repository
where
you
will
find
instruction
on
how
to
install
kubeflow
as
well
as
community
links.
This
is
where
our
links
for
slack
some
of
our
meeting
links
are
there.
Finally,
the
operator
framework,
if
you
want
to
install
qflo
using
operator
as
paul
mentioned,
definitely
try
check
it
out
and
try
it
out.
Finally,
I'd
like
to
thank
you
for
spending
the
last
20
to
30
minutes
listening
to
us
talk.