►
From YouTube: Cloud Native Machine Learning Systems at Day Two and Beyond - Sophie Watson & William Benton
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2021 Virtual from May 4–7, 2021. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Cloud Native Machine Learning Systems at Day Two and Beyond - Sophie Watson & William Benton, Red Hat
You’re probably already convinced that Kubernetes is the right infrastructure for your next machine learning initiative, but you may not be ready for some of the speedbumps that await you on the way. This talk will introduce some of the challenges unique to machine learning systems, prepare you for the tradeoffs you’ll face supporting practitioners and putting systems in production, and present some of the additional tools you’ll need in your DevOps toolbox as your cloud-native machine learning systems mature. You’ll learn how to negotiate pitfalls related to interactive development, reproducibility, and monitoring machine learning systems in production with concrete solutions inspired by our experience with end-users in various industries.
https://sched.co/ekC2
A
B
Hi,
I'm
sophie
watson,
I'm
a
data
scientist
at
red
hat
and
I've
also
been
helping.
Customers
run
their
machine
learning
workflows
on
kubernetes,
so
machine
learning
systems
are
just
distributed
systems
and
so
obviously
kubernetes
provides
the
primitives
to
make
them
manageable.
This
isn't
a
controversial
opinion
anymore.
Everybody
agrees
that
kubernetes
is
now
the
default
choice
for
machine
learning
systems.
B
A
A
So
I
want
to
especially
call
out
this
sort
of
inner
loop
here
where
a
lot
of
the
data
scientists
time
is
spent
and
that's
in
sort
of
feature,
engineering
or
figuring
out
how
to
expose
the
structure
and
data
so
that
a
machine
learning
algorithm
can
exploit
it
model,
training
and
tuning,
which
is
sort
of
using
computing
techniques
to
exploit
that
structure
and
identify
good
trade-offs
in
characterizing
input,
data
and
model
validation,
which
is
verifying
that
the
model
has
actually
generalized
from
the
data.
It's
seen
and
that
we
could
trust
putting
it
into
production.
A
Now
this
inner
loop
has
a
lot
of
iteration
and
experimentation,
and
both
reproducibility
and
velocity
are
going
to
be
really
important
here
and
the
output
of
that
inner
loop
is
going
to
be
a
machine
learning
model.
Now
the
model
gets
a
lot
of
popular
attention.
Even
if
you
don't
spend
a
lot
of
time.
A
But
if
you
haven't
heard
of
this
paper
by
d
scully
and
some
colleagues
at
google
called
hidden
technical
debt
in
machine
learning
systems,
you
should
check
it
out.
This
paper
has
been
read
a
lot
and
cited
probably
even
more
than
it's
been
read,
but
the
argument
is
basically
that
machine
learning
techniques
are
relatively
easy
to
develop,
but
machine
learning
systems
are
relatively
hard
to
maintain.
A
So
some
of
you
here
today
are
probably
old
enough
to
remember
the
y2k
problem
in
which
a
lot
of
software
had
to
be
retooled
because
it
used
date,
formats
with
two
character
dates
to
represent
a
year,
and
so
it
would
only
work
in
a
single
century.
So
once
we
went
from
1999
to
2000,
software
might
think
it
was
in
1900
again
machine
learning
systems
have
analogous
problems
every
single
day,
which
is
why
reproducibility
is
really
important.
We
need
to
be
able
to
make
it
possible
and
easy
to
change.
A
Similarly,
the
library
stack
you're
using
probably
has
a
lot
of
dependencies
on
user
space.
Libraries
that
are
outside
of
your
language
ecosystem,
like
native
libraries,
for
linear
algebra,
on
massive
chunks
of
the
python
or
java
ecosystem,
which
have
a
lot
of
subtle
interdependencies
and
even
on
accelerator
drivers
installed
on
your
host.
A
If
you
need
to
change,
one
of
these
libraries,
for
example,
to
fix
a
security
issue,
provide
some
new
functionality
or
even
just
upgrade
a
transitive
dependency
that
several
things
depend
on.
It
might
perturb
your
entire
stack,
and
so
it's
important
that
we
need
to
be
able
to
roll
back
destructive
changes,
just
as
we
would
with
regular
software,
but
these
problems
are
much
more
frequent
when
we're
dealing
with
a
lot
of
libraries
and
a
lot
of
libraries
that
are
constantly
changing.
A
Another
issue
is
explainability.
If
you're
in
a
regulated
industry,
you
might
have
a
machine
learning
model
making
decisions
about
whether
or
not
to
write
someone
alone.
This
can
be
a
tricky
problem,
even
if
you
get
asked
for
an
explanation
right
away,
but
it's
one
that
practitioners
in
these
industries
plan
for
they'll
explain
how
they
encoded
the
applicants
as
features
and
show
what
aspects
of
the
application
turned
into
inputs
for
the
machine
learning
model
and
show
how
the
model
evaluated
them.
A
This
usually
works
out.
But
what,
if
you
need
to
explain
a
decision
from
a
few
years
ago,
if
you
don't
have
a
way
to
reproduce
that
entire
pipeline,
exactly
as
you
executed
it,
you
won't
be
able
to
explain
your
decision
in
many
organizations,
data
cleaning
and
even
some
future
engineering
steps
are
manual
processes,
but
you'll
need
total,
reproducibility
and
versioning
across
an
entire
pipeline.
A
If
you're
going
to
be
able
to
answer
these
kinds
of
explainability
demands,
another
concern
that
we
all
have
to
deal
with
is
the
gdpr,
and
I
say
we
all
have
to
deal
with
it.
Even
though
it's
an
eu
regulation,
because
any
organization
that
deals
with
eu
citizens
is
bound
by
the
gdpr
under
the
gdpr
users
have
a
right
to
be
forgotten,
which
means
that
you
need
to
be
able
to
show
them.
There's
no
trace
of
their
data
left
in
your
system.
A
B
Thanks
will
so
the
data
science
world
moves
fast,
it's
still
a
new
and
growing
industry,
and
that
brings
with
it
many
challenges
that
kubernetes
just
isn't
equipped
to
handle
out
of
the
box,
and
these
challenges
are
really
exacerbated
by
the
way
in
which
data
scientists
work,
data
scientists,
they're,
used
to
having
total
freedom
and
really
having
to
share
or
publish
their
work
for
them.
The
convenient
thing
is
to
self-manage
an
environment,
so
they
have
total
control.
Think
a
software
developer's
computer
in
1998.
B
In
practice,
these
setups
aren't
conducive
to
rapid
prototyping
and
really
lack
that
flexibility
that
data
scientists
are
used
to
so
diving
into
the
work
of
data
scientists
a
bit
further.
We
as
data
scientists
aren't
focused
on
release
engineering.
When
I
wear
my
data
science
hat,
my
job
is
simply
to
find
meaningful
patterns
in
data
and
left
to
my
own
devices.
B
Expertise
means
that
it's
not
uncommon
for
data
scientists
to
end
up
working
in
a
global
environment
that
contains
every
package
we
ever
used
ever
now
to
achieve
reproducible
and
release
worthy
artifacts.
We
first
need
to
identify
the
minimal
dependencies
of
that
workload
and
once
those
dependencies
have
been
vetted
by
it,
they
can
go
ahead
and
produce
a
certified
image
that
satisfies
the
data
scientists
requirements
whilst
adhering
to
the
company
policy,
but
just
identifying
those
libraries
isn't
always
enough.
B
So
tools
like
source
to
image
and
draft
make
it
really
easy
to
build
these
images
rapidly.
But
as
I.t,
you
might
be
supporting
six
different
data
science
teams,
each
of
which
wants
to
use
a
different
incompatible
version
of
the
same
library
so
unless
you've
got
a
really
effective
vetting
process
and
software
supply
chain.
Your
official
data
science
images
are
going
to
be
laughably
out
of
date
by
the
time
they're
released.
A
A
Eliminating
any
manual
reimplementation
and
automating,
where
possible,
relying
on
computers
is
generally
better
than
relying
on
humans,
but
these
model
microservices
aren't
the
same
as
standard
microservices,
for
example,
with
your
conventional
cloud
native
microservice
you'll
have
an
api
that
accepts
things
that
look
like
program
domain
objects.
For
example.
In
this
case,
we
have
an
employee,
id
and
employee
structure
with
an
employee
id
an
employee
name
and
a
date
range
requesting
time
off
in
a
hr
system,
and
it's
returning
a
tuple
that
has
a
status
of
ok,
but
a
machine
learning
model.
A
A
A
Now
our
model
service
is
going
to
expect
raw
data,
transform
it
into
feature
vectors
and
pass
it
through
the
model
to
make
a
prediction.
But
if
we
expect
a
feature
vector,
our
api
is
just
a
vector
of
floating
point
numbers
and
there's
really
no
way
for
us
to
sort
of
know
how
we
got
to
that
vector
floating
point
numbers.
There
are
all
sorts
of
techniques
we
might
use
to
reduce
large
vectors
to
small
vectors
or
to
reduce
images
or
program
domain
objects
into
vectors
as
well.
A
If
we
had
a
pipeline
service
instead,
we
could
take
in
raw
data,
transform
that
into
feature
vectors
and
make
a
prediction
like
this
and
again
with
perceptual
models.
You
have
the
additional
problem
of
how
you've
actually
encoded
even
an
image,
which
is
structured
data
that
you
could
represent
as
a
vector
of
numbers
into
that
vector
of
numbers,
and
that
might
be
something
you
want
to
capture
as
well.
B
So
we've
talked
about
model
services
and
interacting
with
them,
but
where
do
they
come
from?
There's
not
always
a
simple
way
to
go
from
an
exploratory
data
science
environment
to
a
production
artifact?
And
you
might
end
up
rewriting
or
re-implementing
code,
but
if
you're
doing
that
you're
introducing
possible
errors
and
inconsistencies
between
training
and
deployment.
B
B
B
For
example,
your
builder
pod
might
need
more
resources,
time
memory
and
gpus
in
order
to
train
a
model
and
you're
going
to
have
to
account
for
this
in
your
policies
and
identify
a
cut-off
where
it
no
longer
makes
sense
to
train
models
in
builds
for
some
models
that
require
tens
of
gigabytes
of
ram
days
of
gpu
time
and
that
produce
enormous
outputs.
You
may
need
to
orchestrate
training
without
using
container
builds,
but
you'll
need
to
decide
where
that
trade-off
is
for
your
organization,
and
you
should
definitely
use
the
icd
first.
B
So
monitoring
cloud
native
applications
is
obviously
important,
but
it
becomes
even
more
important
when
we're
talking
about
managing
machine
learning
systems.
So
here
I've
got
a
model
scoring
pipeline.
It
takes
in
data,
transforms
it
and
makes
predictions
now
we
can
check
metrics
about
the
predictions
that
are
made
and
we
can
check
the
standard,
microservice
performance
metrics
as
well
to
monitor
it
and
set
up
alerts
if
something
changes,
notably,
but
finding
out
that
something
has
notably
changed,
doesn't
mean.
We
know
what
has
caused
this
change
or
how
to
identify
it.
B
Changes
in
recorded
metrics
in
the
model
at
the
model
level,
don't
tell
us
exactly
where
in
the
workflow,
something
has
gone
wrong
and
there
are
things
that
can
go
wrong
at
every
stage
of
this
workflow
suppose,
for
example,
your
machine
learning
model
is
detecting
fraud.
It's
been
trained
to
identify
fraudulent
credit
card
transactions.
B
B
Another
example
of
where
things
can
go
wrong
is
in
the
feature,
engineering
stage.
Suppose
you
realize
that
two
columns
in
your
data
set
were
correlated
now.
You
don't
need
both
of
them,
so
you
decide
that
you're
gonna
get
rid
of
one
now
suppose
the
data
source
changes.
Perhaps
someone
else
realized
that
they
don't
need
to
be
collecting
both
pieces
of
information,
so
they
delete
one
of
the
fields
and
they
end
up
deleting
the
one
that
you
didn't.
B
When
standard
software
breaks,
it
often
fails
to
compile-
or
it
returns
an
error
message,
but
machine
learning
models
don't
fail
in
the
same
way,
they're
still
going
to
carry
on
making
predictions,
but
these
predictions
are
going
to
be
increasingly
wrong.
More
often,
another
example
is
in
the
medical
industry.
Suppose
you
trained
a
neural
language
model
on.
B
Supposedly
trained
an
nlp
model
on
medical
journals
back
in
january
2020,
it
might
give
you
a
word
cloud
that
looks
something
like
this
now.
You
might
have
heard
of
this
thing
called
the
novel
coronavirus
which
has
been
causing
quite
an
impact
in
metal
medical
journals
in
the
middle
part
of
2020,
but
no
models
trained
last
year
or
early
in
this
year
will
even
know
that
it
exists
so
monitoring
your
models,
identifying
changes,
it's
all
well
well
and
good,
but
it's
not
really
sufficient
to
solve
your
problems
now.
B
B
A
Thanks
sophie,
let's
look
at
the
big
picture.
Now:
we've
seen
how
the
machine
learning
workflow,
how
practitioners
build
machine
learning
systems,
but
we
haven't
seen
what
a
machine
learning
system
actually
looks
like
in
production
and
how
those
individual
parts
that
individual
humans
work
on
fit
together
to
get
a
sense
for
the
territory.
Let's
take
a
look
at
a
map.
A
First,
let's
look
at
what
data
engineers
do.
Data
engineers
are
responsible
for
making
data
at
rest
and
data
in
motion,
both
structured
and
unstructured
accessible
to
machine
learning
systems,
and
they
thus
need
to
manage
multiple
system
services
to
provide
access
to
data
and
make
it
easier
for
machine
learning,
workflows
and
systems
to
consume
federated
data
from
multiple
sources.
A
These
roles
overlap
too
in
key
places.
Machine
learning
systems
model
some
aspect
of
the
real
world,
which
is
a
complex
and
changing
place
by
incorporating
machine,
readable
feedback
about
historical
predictions.
Our
systems
can
improve
with
time
by
identifying
mispredictions
they've
made
in
the
past,
as
practitioners
learn
more
about
aspects
of
the
world
that
they're
modeling,
their
insight
will
lead
to
improved
techniques
as
well,
but
only
if
we
can
enable
them
to
iterate
with
high
velocity.
A
Nearly
every
part
of
the
system
will
produce
metrics
reports
for
model
validation,
summary
statistics
and
histograms
for
data
sources,
conventional
metrics
for
conventional
application,
components
and
so
on,
but
some
components
will
consume
metrics
as
well
as
sophie
mentioned.
Drift
detection
is
an
important
problem
we
need
to
address
and
we
can
use
the
kubernetes
monitoring
stack
to
support
drift
and
outlier
detection
to
identify
problems
proactively
with
our
models
and
pipelines.
A
B
We
talked
about
how
there's
a
lot
of
heavy
lifting
done
by
this
particular
part
of
the
workflow.
That
path
from
the
developed
model
to
the
deployed
model
is
not
as
straightforward
as
one
would
like,
and
we
discussed
how
tools
like
source
to
image
and
good
ci
cd
practices
can
help
to
ease
this
transition,
and
once
the
model
is
in
production,
we've
still
got
these
unique
challenges
that
machine
learning,
microservices
face
and
regular
microservices
do
not,
for
example,
managing
complex
software
stacks
and
managing
pipeline
apis
versus
model
apis.
A
The
second
is
a
paper
about
why
machine
learning
systems
are
really
a
lot
like
distributed
systems
in
a
way
that
enables
them
to
benefit
from
kubernetes,
and
it
introduces
a
concept
called
intelligent
applications
to
make
sense
of
machine
learning
systems
in
the
cloud
native
era.
Thanks
again
for
your
time,
neither
of
us
is
speaking
for
our
employers
in
this
case,
so
the
best
way
to
reach
us
is
through
our
twitter
handles
which
are
at
the
bottom
of
the
slide.