►
From YouTube: The Challenges Managing a Kubernetes-Based Machine...- Yuzhui Liu & Keith Laban, Ed Shee, Keshi Dai
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2023 in Amsterdam, The Netherlands from April 17-21. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
The Challenges Managing a Kubernetes-Based Machine Learning Infrastructure - Yuzhui Liu & Keith Laban, Bloomberg; Ed Shee, Seldon; Keshi Dai, Spotify
Speakers: Keith, Keshi Dai, Yuzhui Liu, Ed Shee
Managing a machine learning infrastructure is a great challenge, as its scope covers both common infrastructure tasks – such as cluster management, network, security, container management, and observability – and ML-focused tasks – such as GPU compute, data exploration, distributed training, and model serving. Kubernetes and its prosperous open source ecosystem provides great infrastructure tools (e.g., Knative, Cloud Native Buildpacks, Argo, and Envoy), as well as ML-focused projects (e.g., Kubeflow, KServe, Seldon Core, and KubeRay) that enable infrastructure engineers to build a modern machine learning infrastructure. In this panel, you’ll hear from engineers at Bloomberg, Seldon, and Spotify about how they’re using the Kubernetes ecosystem to provide machine learning infrastructure and their current challenges. Panelists represent a variety of use cases, including end-users and infrastructure providers, as well as both on-prem and cloud-based infrastructures.
A
Welcome
everyone
this.
This
is
a
panel
discussion,
so
we're
going
to
try
and
keep
it
as
interactive
as
possible,
it's
better
for
us
and
better
for
you
guys
if
yeah,
if
you
ask
our
staff
and
and
we
kind
of
bounce
off
you
as
well,
rather
than
just
doing
a
monologue
between
the
four
of
us
just
before
we
get
started
and
do
introductions
just
to
kind
of
gauge
the
audience
a
bit.
A
A
B
A
A
C
The
team
leader
of
data
science
around
time
from
Bloomberg,
so
we
provide
on-prem
biometer
note
based
machine
learning
infrastructure.
We
provide
multiple
products
through
our
platform,
including,
like
notebook,
training
model,
serving
spark
extra,
and
we
have
many
very
interesting
internal
use.
Cases
such
as
machine
learning
for
financial
product
for
our
news,
product
and
extra.
A
So
I'm
Ed
Ed,
sheer
I'm,
head
of
developer
relations
at
a
company
called
Selden,
and
you
know
we're
slightly
different
from
Bloomberg
and
Spotify
and
that
we
build
machine
learning
software
that
we
then
you
know,
give
to
other
customers
to
deploy
stuff
on
and
we
do
the
like
the
last
mile
of
machine
learning.
So
we
do
deployment
monitoring
and
explainability
side
of
things
but
work
across
all
sorts
of
use.
Cases
which
is
kind
of
fun
to
see
yep.
D
Hey
my
name's
sketchy
I'm,
an
engineer
on
the
MLP
team
at
Spotify
I
mean
ml
is
almost
the
everything
we
do
at
Spotify
and
people
like
teams,
using
machine
learning
for
music
recommendations
and
discover
new
musics
and
there's
a
lot
of
all
kinds
of
applications
for
ML
like
bring
in
storage
from
small
like
so
provide.
The
learning
on
the
type
of
data
sets
are
to
very
complex,
deep
learning
and
on
the
graphic
neural
networks.
D
B
You
know
machine
learning,
focus
problems
to
to
address
a
lot
of
the
data
problems
at
Bloomberg
from
natural
language
to
Market
data
across
a
diverse
set
of
teams
at
the
company,
and
we
ran
into
a
lot
of
infrastructure
related
challenges
in
building
our
ml
solution,
and
you
know
along
the
way.
We
also
realized
that
the
tools
that
we're
building
so
we
were
building
you
know,
training
tools,
imprints
tools,
for
you
know
going
into
production
tools
for
data
transformation,
so
using
spark
workflow
tools
to
solve.
B
You
know
machine
learning,
problems
Jupiter
notebooks,
for
for
your
data
exploration.
We
we
started
realizing
like
actually
the
tools,
we're
building
yeah,
it's
useful
for
machine
learning,
but
it
also
actually
has
like
a
lot
of
other
applications
to
it,
and
so
you
know
I
think
for
us,
like
over
our
journey.
B
A
Yeah
I
think
actually
I'm
going
to
ask
another
question
to
the
audience
quickly
here,
so
I'm
keeping
your
hands
up.
If
you
are
a
data
scientist
right,
if
that's
your
title,
yeah,
like
one
person
in
the
horror,
two
okay,
one
yeah
yeah
so
I-
think
that's
kind
of
like
part
of
the
problem
here
is
that
yeah?
Unsurprisingly
right,
we
don't
have
tons
of
data
scientists
here
at
kubecon,
but
ultimately
we're
kind
of
building
these
platforms
for
data
scientists
right
so.
B
C
C
What's
the
user
group
like,
of
course,
primarily
for
data
scientists
and
the
Machine
learning
engineer,
who
want
to
run
large
compute
workload
and
surf
model,
but
at
the
same
time,
especially
for
notebook,
there
are
a
lot
of
no
engineering
use
cases
as
well
like
want
to
do
a
report.
I
want
to
do
some
data
analysis.
Do
some
Key
Club
spark
to
do
some
ETL?
So
that's
a
that's
a
that's
something
that
we
have
internally
discussed
a
lot
I
wonder
how
other
company
think
about
it.
D
Yeah,
that's
definitely
true
at
Spotify
as
well
like
the
ml
problem
as
Spotify
is
the
Department
of
long
tail
like
all
kinds
of
different
ml
applications.
We
have
very
diverse
ml
practitioners
using
ml
in
very
different
ways.
People
need
the
notebooks
people
need
pipelines.
People
need
to
be
reliable
way
to
conduct
their
the
ml
continuously.
So
when
we
design
the
platform,
it's
really
neat
to
think
about
layers
like
say
how
about
we
just
can
get
an
API,
so
can
streamline
some
standardized
process.
D
So,
when
easy
get
on
our
platform,
they
can
very
quickly
to
build
a
prototype
and
build
ml
application
through
a
few
steps.
But
if
their
application
really
goes
deep,
you
really
need
a
lot
of
customizations.
Our
platform
also
needs
also
need
to
have
the
flexibility
to
allow
them
to
like
using
the
low
level
API
to
use
kubernetes
to
use
computer
Powers.
So
they
can
build
a
very
good
application.
Yeah
about.
A
B
On
what
they
want
to
do,
yeah
yeah!
That's
that's
an
interesting
way
of
levels
right
like
so.
Our
myorg
is
called
the
runtime
team,
so
we've
been
really
focused
on
the
level
of
exposing
up
to
sort
of
the
technology
layer
of
the
runtime.
So
if
it's
Spark
It's,
how
do
you
run
spark
on
kubernetes
as
good
as
possible?
B
There
might
be
a
number
of
products
that
you
expose
on
using
spark
because
spark
turns
out
to
be
really
good
for
reporting
functionality
and-
and
you
know,
analytics
tools,
but
it
also
happens
to
be
an
extremely
important
project
for
amp
or
part
of
the
ml
life
cycle.
The
the
data
engineering
right,
and
so
you
might
want
to
have
spark
as
part
of
a
data
science
product.
You
know
platform
in
one
UI,
but
you
know
at
a
sufficiently
large
company.
You
probably
have
plenty
of
other
Engineers
somewhere
else.
B
That
also
want
to
use
spark
for
much
much
different
things,
and
so
you
know
the
direction
that
we've
sort
of
taken
is
like.
How
do
we
build
something
flexible
enough
that
you
know
other
platforms
can
platform
eyes
on
top
of
us
without
having
to
reinvent
the
Wheel
from
from
the
ground
down
so
I'm
curious
in
the
audience
who
thinks
they
have
too
many
controls
to
too
much
there's,
not
enough
abstraction,
no,
no
hands.
Okay.
B
A
No
I
think
I
think
which
it's
a
lost
session
of
the
day.
Vibes
right
people
we're
trying
to
keep
you
awake.
We
do
actually
have
a
stooge
planted
with
a
mic
by
the
way.
So
at
any
point,
if
you
want
to
chip
into
the
conversation,
please
just
stick
your
hand
up
and
and
yeah
you
can.
You
can
get
involved,
but
we
actually
have
a
question
over
there.
A
So
yeah
do
you
want
to
go
over
there
while
you're
doing
that
I
was
going
to
say
yeah
for
our
like
deployment
platform,
we
build,
we
tend
to
see
kind
of
like
three
types
of
users.
That's
like
your
I,
don't
know
what
kubernetes
is
and
I
don't
know
need
to
know
what
it
is
type
user
and
for
them
there's
like
a
you,
know,
lovely
UI.
You
can
just
walk
through
and
drag
and
drop
and
kind
of
you
know,
get
your
model
deployed
and
then
there's
like
your
kind
of
python
user.
A
Who's
like
just
wants
to
use
the
SDK
and
have
it
deployed
again
with
a
few
config
params,
and
then
there's
like
your
kind
of
super
user
who
understands
Cube,
wants
to
go,
and
you
know,
play
with
the
container
spec
and
do
all
sorts
of
things
underneath,
and
we
feel
we
have
to
account
for
all
of
those
right
and
actually
the
the
more
customers
use
us
the
more
they
end
up
down
in
that.
Like
super
user
type
thing.
D
Yeah
definitely
had
a
Spotify
supporting
is
a
big
theme
in
the
platform
team
like
since
we
have
all
kinds
of
users
they're
using
all
kinds
of
tools.
How
do
we
provide
different
level
of
support
to
enable
them
to
empower
them
to
build
ml
application?
It's
very
challenging
our
plan
for
team
has
a
user
engagement
team,
so-called
they
basically
gonna
bootstrap
the
ml
applications
in
the
future
team,
from
zero
to
one
so
they're
gonna
go
out
club
with
different
teams
and
teach
them
how
to
use
our
tools
and
to
build
applications.
D
B
Saying
yeah
we
I
mean
we
don't
have
a
user
engagement
team
but
I
I
kind
of
wish.
We
did
we
we
put
a
lot
of
energy
into
documentation
and
internal
training,
and
you
know:
we've
been
quite
successful
with
it
but
again
like,
like
you
said,
there's
there's
sometimes
a
misstep
in
in
where
the
level
of
abstraction
actually
is
and
and
covering
all
of
those
bases
sometimes
gets
really
hard
and
so
I.
Imagine
your
user
engagement
team
can
kind
of
step
in
and
do
some.
You
know
Solutions
engineering
with
them
yeah.
Definitely.
D
I
mean
the
talk
is
very
important,
but
dark
can
only
like
cover
only
can
help
you
like
to
certain
extents.
If,
like
users
are
ready
to
go
lower
level,
say
Hey,
how
do
I
build
a
custom
component
to
doing
using
a
different
framework?
Then
you
really
need
people
with
the
expertise
to
help
them
to
understand
the
lower
level,
library
and
low
level.
Sdk.
A
Yeah
I
mean
you
think
right
as
a
software
vendor
this,
but
it
should
be
easier,
but
and
if
anything,
it's
almost
harder
right
because
you
know
being
on
top
of
kubernetes.
You
have
this
like
wealth
of
cloud
native
tooling,
that
you
can
use
all
these
fantastic
cncf
projects
that
probably
all
of
us
around
here
are
using
for
for
things.
A
You
know
all
over
our
organizations,
but
that
you
know
like
support
and
education
window
suddenly
becomes
absolutely
enormous,
because
you
know
you're
using
a
service
mesh
right
as
part
of
your
product
and
people
are
then
asking
you
I
have
an
issue
with
that.
It's
conflicting
with
something
else
on
my
cluster.
Is
that
now
our
job
to
to
help
them
with
it,
and
it
becomes
really
difficult
right
all
of
these
kind
of
things.
A
We
don't
know
we
don't
have
like
a
specific
tool.
I
guess
you
know
again,
one
of
the
great
things
about
being
a
vendor
is,
at
least
you
know.
We
can.
We
have
like
a
paid
support
option
and
people
can
go
through
that
and
then
you
have
dedicated
people.
Who
can
you
know
who
have
skill
sets
and
all
these
cncf
tools
that
we
use
but
yeah
there's?
No,
like
you
know
magic,
you
know,
run
the
debug
wizard
type
thing.
Unfortunately,.
C
One
thing
we
always
is
that
what's
the
battery
between,
like
user
responsibility
and
infrastructure,
responsibility
and
I
think
that's
always
a
huge
challenge
for
managed
infrastructure,
basically
any
type
of
managed
infrastructure,
and
we
ask
ourselves
like
do
we
provide
enough
monitoring?
Do
we
have
enough
metrics
enough
logging
like
sufficient
dashboards?
C
So
we
can
offload
some
of
those
support,
turn
it
into
self-debugging
tools
and
documentations,
but
then
we
always
found
It's,
never
enough,
because
the
user
base
is
very
different,
like
with
more
stronger
engineering
background
like
having
like
CPU
metrics
memory,
metrics
very
helpful
for
those
group
of
users,
but
for
a
different
group
of
users,
maybe
focus
more
on
data
science
or
maybe
focus
more
on
data
analysis
like
this
kind
of
tool,
is
not
like
the
right
tool
for
them
to
do.
Self-Debugging
now,.
E
But
when
you
try
to
scale
it
over
your
entire
fleet
around
the
world,
as
most
people
here
probably
know
you
run
into
difficulties.
So
how
do
you
establish
a
culture
both
in
management
and
within
the
group
you
mentioned?
Documentation
is
a
big
thing
at
Bloomberg,
I'm
sure
that
culture
didn't
come
out
overnight
and
others
have
mentioned
other
things
at
your
respective
companies
that
have
been
established.
How
did
you
go
about
establishing
them,
making
that
relationship
with
upper
management,
c-suite
and
kind
of
having
that
trickle
down
to
the
rest
of
the
company
before.
E
Kind
of
supporting
and
scaling
models,
supporting
and
scaling
training
building
tooling
for
the
data
scientists.
Basically,
all
the
things
that
you
guys
mentioned
is
how
we're
defining
it,
but
just
like
how
you
said
earlier:
ml
software
data
engineer,
ml
Ops
devops,
whatever
hat
we
kind
of
have
to
wear
there
but
yeah.
D
I
gotta
try
to
take
a
stab
at
Spotify.
We
have
called
ml
golden
pass,
which
kind
of
like
a
standardized
way
to
do.
Ml
and
our
platform
also
viewed
those
like
standard
ml
Ops
say
if
you
want
to
do
the
like
feature
processing
we
have
component
for
you
see.
If
you
want
to
do
a
data
validation,
then
that's
component
for
you,
and
our
platform
also
requires
some
like
standardization
or
the
future
schema
and
evaluation
metrics,
so
trying
to
standardize
the
whole
ml
workflow
into
a
sequence
of
components.
A
Something
we
use
is
is
called
like
the
the
V2
inference,
protocol
or
open
inference,
protocol
or
whatever
I
know
that
Bloomberg
use
it
as
well
right
and
that,
despite
using
a
different
deployment
engine
underneath
right
and
that's
something-
that's
really
cool
because
it
means
like
you
know
it.
If
I
want
to
interact
with
a
machine
learning
model,
you
know
that's
been
deployed
on
Selden
right,
it's
the
same
interface
that
I'm,
using
as
if
I'm
you
know,
I
suddenly
start
interacting
with
Bloomberg's
models
right
and
so
standardizing.
A
Those
things
makes
it
not
just
user
easier
for
the
the
people
who
have
to
deploy
these
things,
but
actually
for
people
who
have
to
work
with
it
as
well.
You
know
the
software
engineers
and
the
application
owners
who
are
integrating
with
those
apis
as
well
so
yeah
that
that
kind
of
thing
definitely
helps.
C
C
Oh
sorry,
can
you
hear
me
yeah?
This
I
would
like
to
mention
that
one
thing
that
we
are
like
within
Bloomberg.
We
are
trying
to
establish
more
streamlined
more
for
our
data
science
platforms
that
to
Define
over
escalation
process
better,
like
we
support
like
like
over
a
thousand,
we
have
more
than
one
thousand
people
just
in
our
support
chat
and
we
have
100
teams
on
board
into
the
platform
so
to
define
a
way
to
escalate
issues.
C
B
Yeah
I
think
a
lot
of
it.
You
know
documentation
golden
paths,
all
that
stuff
is
really
great
and
and
something
we
think
about
as
as
well
but,
like
usually
said,
like
really
providing
transparency
to
you
know
your
upper
manage
management,
senior
management
into
all
the
work
you're
actually
doing,
is
extremely
hard
right
and
finding
ways
to
to
do
that.
B
Like
there's,
not
one
answer
to
how
to
do
that
and
then
I,
don't
I,
don't
know
where
you
are
in
your
in
your
journey
towards
this,
whatever
you
want
to
call
it,
but
there's
also
kind
of
the
subtext
of
just
like
Cloud
native
mentality
in
your
company.
So
if
you're
in
a
fairly
mature
company,
just
you
know,
shifting
to
Cloud
native
before
you
even
introduce
that
ml
stuff
to
the
conversation
is
potentially
a
challenge
too.
B
F
A
I
I
mean
I
can
take
a
style
of
this,
while
you
guys
think
about
it
so
from
from
my
perspective,
we're
about
to
go
through
this
actually,
so
we
have
a
like
a
major
release
of
our
platform.
That's
coming
that's
going
to
allow
people
to
when
they
create
that,
like
custom
python
models,
it's
it's
going
to
be
slightly
different
right
now.
A
I
think
that's
really
important,
because
even
if
there
are
tons
of
benefits
to
doing
something
a
new
way
right,
the
fact
that
you've
broken
the
way
they
did
stuff
you
have
to
give
them
time
and
then
provide
like
as
much
education
as
we
can
firstly
like
why
they
should
move
to
the
newer
method,
but
also
to
make
it
as
easy
as
possible.
And
that's
you
know.
A
The
great
thing
is:
that's
literally
my
role
at
the
company,
so
you
know
it's
to
create
videos
and
tutorials
and
blogs,
and
things
like
that
to
to
make
it
as
easy
as
possible.
So
it's
like
okay,
you're
upgrading
from
this
version
to
this
version.
Your
custom
python
model
is
going
to
change
it
a
little
bit.
This
is
exactly
what
you
need
to
do.
Don't
worry
too
much.
Etc.
B
A
B
Think
setting
expectations
is
just
is
so
important
and
I
think
that's
something
the
kubernetes
community
I
think
as
a
whole
has
done
a
really
good
job
of
in
terms
of
labeling
things
as
Alpha,
Beta
and
so
on
and
and
they're
they're
really
clear
about
what
that
actually
means
in
terms
of
interfaces.
Stability,
things
like
that
and
so
kind
of
following
those
guidelines
as
a
whole.
D
Yeah
for
us
I
guess
on
communication
and
collaboration
are
the
keys
like
wherever
we
do,
the
big
upgrades
we
let
users
know
like
a
Miles
Ahead
just
tell
them,
you
need
to
expect
breaking
changes,
and
then
we
also
put
a
lot
of
effort
to
write
in
the
migration
dock.
So
you
can
follow
step
by
step
to
update
their
Pipeline
and
my
workflows,
and
we
also
have
the
migration
Force.
Basically
we're
going
to
help
them
to
migrate
their
pipeline
to
a
certain
degree.
G
So
I'm
I'm
curious
what
your
experience
has
been
at
your
various
companies
with
maturing
and
machine
learning
organization
into
something
that's
sane
and
operational.
You
know
at
our
company,
we
you
know,
AIML
is
a
relatively
new
thing.
I
know
a
lot
of
companies,
that's
been
around
for
ages
and
they
already
know
how
to
do
it.
But
for
us
we've
like
acquired
companies
that
you
know
have
a
AI
component,
and
so
you
have
a
bunch
of
you
know:
data
scientists
that
have
systems
that
they
built
themselves
their
pets.
They
don't
want
to.
G
Let
go
of
them,
they
don't
understand,
they
don't
want
to
play
with
anyone
else.
They
don't
want
to
hand
this
over
to
I.T
or
to
a
devops
or
as
SRE
organization,
and
so
you
know
you
mentioned
things
like
golden
paths
and
stuff
as
a
way
of
kind
of
building
that
culture.
But
do
you
have
any
like
War,
Stories
or
just
advice
for
how
does
in
you
know,
an
I.T
or
an
infrastructure
organization
interact
with
teams
like
that
to
help
bring
them
on
board.
C
That's
a
very
good
question,
so
I
would
say,
especially
in
a
large
company.
I
found
it's
hard
to
have
one
solution
to
fit
fit
all
so
I've
definitely
see
there
are
things
want
to
build
some
their
own
infrastructure
or
maybe
part
of
it
and
then
run
another
part
of
it
in
our
platform?
So
I
think
that's
I,
think
that's
actually
common,
because
I
believe
there's
no
one
solution
fit
all,
but
I
think
as
some
features
material
like
and
we
found
out.
B
Like
the
problem,
I
think
gets
better
over
time
if
you
find
the
right
set
of
users
in
the
beginning
in
the
company
that
are
going
to
be
strong
advocates
for
the
platform
You're
Building.
Eventually,
you
know
you,
you
might
get
some
of
the
rest
of
that
market
share
and
you
might
not.
It
was
definitely
kind
of
a
big
challenge.
When
we
first
started
everyone
had
their
own.
B
You
know
metal,
whatever
Big
Iron
machines
gpus,
whatever
that
they
can
do
whatever
they
want
it
on
and
then
when
it
broke,
and
there
was
no
one
there
to
fix
it.
They
realized.
Oh
wait.
I'm,
not
you
know
doing
data
science
I'm
not
doing
this
ml
research,
anymore,
I'm,
actually
now
in
SRE
for
some
machine,
and
so
once
that
sort
of
resourcing
dried
up
for
for
those
types
of
people
and
there's
more
messaging
towards
hey.
B
You
know
there's
another
platform
that
you
can
use
where
you
don't
have
to
think
about
these
types
of
issues
anymore.
Then
then
you
know
you
start
start
picking
up
some
of
those
that
market
share,
but
but
it
definitely
takes
you
know
a
long
time
to
build
a
mature
system
where
people
actually
have
that
faith
that
they
can
move
over
to
it.
D
Yeah
definitely,
we
also
cannot
like
rely
on
the
open
source.
Tooling.
The
new
technology,
for
example,
Spotify,
is
investing
in
Gray
like
generic
distributed
processing
system
for
ML
and
to
complement
the
opinionated
ml
workflow.
We
have
for
the
production
systems
so,
which
means
and
my
researchers
and
data
scientists
they
can
access
more
broad
ml
ecosystems
to
use
more
libraries
and
doing
experiments
on
Ray,
which
that's
another
another
aspect,
just
offering
like
more
flexible
online
platform.
So
people
can
do
more
things
on
like
by
themselves.
H
Yeah
over
here,
hello,
so
given
most
of
us,
probably
in
the
room,
are
platform
Engineers,
where
we
may
have
varying
levels
of
understanding
of
the
needs
of
ml
Engineers
data
scientists
Etc
well.
To
be
frank,
for
me,
it's
very
little
I
guess.
My
question
is:
where
should
someone
like
ourselves
get
started
in
terms
of
best
helping
to
understand
the
needs.
A
A
That's
a
that's
a
tough
question
right
because
I
think
it
it
slightly
depends
on
your
size
of
organization,
I.
Think
if
you're
a
a
small
company
where
you're
you
know
one
of
the
only
platform
engineers
and
you
just
have
to
support
these
ml
workloads,
you
might
have
to
be
a
bit
of
a
hero
right
and
and
learn
some
of
this
stuff
yourself,
and
you
know
and
work
closely
with
your
data
scientists,
ml
Engineers,
whatever
you're
calling
them
right.
A
If
that's
the
case
right,
the
you
know
there's
now.
The
good
thing
is
that,
like
tons
of
online
communities
around
ml
ml
Ops
ml
engineering,
you
know
there's
loads
of
resources
and
people
who
can
share
best
practices
and
experiences
because
I
think
we're
you
know
in
this
mlops
world
right
whatever.
That
means
we're
kind
of
you
know
where
devops
was
10
15
years
ago,
and
obviously
it's
accelerated
way
more
quickly,
because
we
can
just
reuse
all
the
best
practices
from
devops,
but
there
are
still
a
ton
of
things
that
are
different.
A
You
know
like
if
you're
a
a
devops
engineer
you
you
won't
have
heard
of
drift
detection
or
why
you
need
to
do
that
in
a
machine
learning
model
and
that's
where
I
was
going
to
say
so,
if
you're,
a
bigger
organization,
I
think
you
can
probably
hire
like
people
to
do
ml,
Ops,
engineering
or
or
machine
learning
Engineers,
who
are
kind
of
closer
to
the
infrastructure
side
and
can
understand
both
sides
of
the
fence,
and
then
they
can.
They
can
kind
of
bridge
the
gap
between
two
teams.
A
B
Well,
I
was
gonna,
go
back
to
something
you
said
earlier,
actually
about
kind
of
the
cncf
landscape
as
a
whole.
You
were
kind
of
hinting
towards
that
and
you
know
that's
just
a
great
resource
to
begin
with
right,
like
there's
so
many
platforms
like
I'm
sure
you're
all
here,
going
to
all
the
talks
of
you
know,
meta
flow
and
kubeflow
and
all
the
different
flows
right
and
then
there
is
all
the
SAS
providers
as
well,
and
so
there
there's
tons
of
solutions
that
you
can
look
at
that.
You
know
provide
some
out
of
the
box.
B
B
You
know
types
of
people,
on
the
other
hand,
at
Bloomberg,
you
know
we're
all
we're
all
on-prem.
We
have
really
sensitive
data,
so
we
don't
we're
and
we're
a
larger
organization.
So
we
tend
to
build.
You
know
a
lot
of
it
ourselves,
but
that
also
takes
a
lot
of
you
know,
time
and
energy
and
and
Investments.
D
Yeah
for
Spotify,
we
run
everything
on
Google
Cloud,
so
we
get
like
a
lot
of
benefits.
We
bootstrap
our
ml
platform
right
on
tensorflow
extended.
It's
like
component
based
approach.
You
find
like
Define
standard
amount
of
suit
component,
so
you
can
round
entire
and
ml
pipelines
on
kubernetes
that
helps
a
lot
to
like
ensure
user
can
like
and
design.
Workflow
then
make
sure
the
workflow
has
everything
you
need
to
develop
a
successful
ml
application.
I
Yeah,
so
I
wanted
to
go
back
to
something
you're
talking
about
earlier
with
debugging,
specifically
about
what
we
expect
people
to
know
about
kubernetes.
So
it
seems
like
we
want
to
have
a
lot
of
abstractions
and
maybe
a
data
scientist.
Maybe
we
don't.
Maybe
they
don't
need
to
know
what
a
pod
is
even
right
at
some
point
but
on
the
same
hand,
we've
been
thinking
a
lot
about
debugging
and
how
to
expose
that
to
the
data
scientist
to
better
understand
their
system
and
sometimes
the
errors.
I
Look
like
a
proxy
couldn't
connect
to
the
Pod
because
the
pods
and
killed
so
they
need
to
understand
what
proxy
is
or
maybe
their
vision's
not
ready.
So
what's
a
replica
set
these
things
kind
of
end
up
leaking
into
the
errors
that
we
expose
to
them?
So
my
question
is:
maybe
specifically,
what
do
you
guys
expect
data
scientists
at
this
point
to
know
and
about
kubernetes
details
and,
if
so,
how
or
how
does
that
leaken
to
your
debugging
platform
and
the
errors
that
you
show
and
exposed
to
them?.
A
We
need
to
have
a
way
to
just
surface
that
there
is
an
error
and
if
it's
something
that's
like,
oh
your,
you
know,
your
model
was
I,
don't
know
like
the
wrong.
You
selected
the
wrong
model
type
right.
You
gave
us
a
tensorflow
model
and
it
was
pytorch
right.
That's
something
we
can
easily
like
tell
them.
If
it's
something
much
lower
level,
we
probably
need
to
tell
them.
There's
an
error,
but
then
have
a
way
to
to
have
that
surface
to
the
right
person.
B
Well,
we
try,
we
try
to
abstract
them
and
thanks
Alexa
by
the
way
it's
a
lecture.
She
has
a
great
podcast
called
Alexa's
input.
If
you
don't
know
her
check
it
out,
you
know
we
try.
We
try
to
set
up
these
abstractions
right
like
that's,
that's
why
we
are
talking
about
personas
and
you
know,
building
the
right
product
for
the
right
people,
but
you
forget
something
and
then
some
pot
and
some
log
kind
of
leaks
through
and
and
you
know,
trying
to
figure
out
what
that
expectation
is
and
like
do.
B
We
have
the
time
to
radically
change
our
assumptions
of
how
we
built
the
product
to
fix
that
specific
type
of
error
message
in
a
way
that
has
some
clear
remediation
for
an
end
user,
and
you
know
the
truth
is
like
these
tools
are
built
on.
You
know
an
extremely
complicated
stack-
and
you
hinted
at
this
earlier
right,
like
are
you
using
a
service
mesh?
B
Are
you
using
istio
link
or
D,
or
you
know,
are
you
using
you
know
case
serve
which
also
brings
in
K
native
into
your
stack
right,
and
so
there
there
are
just
like
a
lot
of
you
know
tools
in
the
middle
to
to
provide
you
know
to
an
end
user.
What's
a
fairly
trivial
thing
like
I
just
want
to
deploy
my
model,
but
it
turns
out
there's
a
whole
lot
of
infrastructure
that
has
to
go
behind
that,
and
so
you
know
a
little
bit.
B
I
think
is
setting
expectations,
and
so
that's
where,
like
ml
Ops,
comes
into
the
equation,
you
know
do
have
people
sitting
with
the
ml
Engineers
who
are
productionizing
the
models
that
can
speak
the
language
of
kubernetes
a
little
bit
better
or
you
know,
is
that
on
the
on
the
runtimes.
Is
that
on
the
platform
team
to
completely
abstract
all
that
stuff
away?
And
it's
real
I
think
it's
a
really
hard
question
to
answer.
A
C
I
can
so
we
actually
use
kubernetes
like
namespace
for
user
isolation.
We
we
choose
to
build
over
a
cluster
in
a
multi-tenancy
way
and
that's
a
very
important
online
infrastructure.
We
have
and
we
isolate,
isolate
different
users
like
credentials,
identities,
resource
code,
this
kind
of
thing
and
allow
them
to
basically
almost
share
of
our
clusters
and
around
the
kick
off
the
workload.
D
We
did
a
similar
way
every
team
they
have
their
own
namespace,
then
the
way
it
works.
We
have
our
own
controller,
like
team,
submit
config
file
Define
like
what
kind
of
service
comes
you
use.
What's
the
owner
of
the
namespace,
like
our
controller,
can
automatically
set
up
those
permissions
for
them
and
make
sure
all
the
parts
running
online
namespace
has
access
to
data?
Has
access
to
gcp
service,
running
on
different
user
projects.
A
Cool
yeah
and
we
we
do
a
little
bit
of
both.
Actually
we
have
like
name
spaces,
and
then
we
also
have
an
abstraction
called
projects
over
the
top
which
allow
people
to
you
know
things
like
models
registered
in
a
catalog
to
be
shared
between
teams
and
things
without
them
necessarily
having
to
have
access
to
the
same
name
space
as
someone
else.
Things
like
that
yeah
I
I
think
we're
pretty
much
out
of
time.
A
You
know
how
to
learn
the
right
resources.
You
know
for
machine
learning,
if
you're
coming
from
the
you
know,
Cube
platform,
engineer,
side
and
I'd
say
just
try
as
best
as
you
can,
if
you're
interested
in
the
space
learn
both
right,
because
whilst
it's
a
hell
of
a
lot
to
learn,
you'll,
never
learn
everything.
A
C
I'll
definitely
suggest
to
embrace
the
community,
so
I
start
to
work
on
my
infrastructure
about
three
years
ago
and
I
have
no
idea
what
kubernetes
is
and
so
throughout
the
past
three
years
being
involved
in
open
source,
Community
being
a
contributor
or
sometimes
just
country.
Sometimes
even
just
contributes
very
small
things.
Pick
up.
A
small
GitHub
issue.
Have
a
discussion
just
learn
a
lot
and,
along
the
way,
I
get
to
know
many.
D
Yeah
I
will
probably,
if
you
want
to
build
an
ml
platform
I
really
like
the
exam
pattern
called
like
Progressive
complexity
exposure.
So
you
need
to
offer
some
like
Streamlight
process
like
ml
golden
pass,
allow
people
to
quickly
start
the
ml
workflow,
where,
if
they
have
more
needs,
they
can
and
like
dive
into
the
details.
Look
at
the
low
level
apis
low-level
docs
to
really
customize
their
applications.
B
Could
be
yeah
as
an
infrastructure
provider?
I
guess
be
really
intentional
about
what
you're,
building
and
set
expectations
for
for
your
users,
especially
in
the
ml
space,
I.
Think
in
the
early
days
of
our
of
our
building
our
platform,
it's
it's
extremely
easy
to
spread
yourself
too
thin,
because
the
amount
of
things
that
you
need
to
build
just
to
get
a
small
ml
project
off
the
ground
is
is
basically
like
go
to
the
solution
center.