►
From YouTube: Exploring ML Model Serving with KServe (with fun drawings) - Alexa Nicole Griffith, Bloomberg
Description
Exploring ML Model Serving with KServe (with fun drawings) - Alexa Nicole Griffith, Bloomberg
KServe (formerly known as KFServing) provides an easy-to-use platform for deploying machine learning (ML) models. KServe is built on top of Kubernetes and provides performant, high abstraction interfaces that allow data scientists to spend more time focusing on building new models, and less time worrying about the underlying infrastructure. This open source project provides a simple, pluggable solution for common infrastructure issues with inference models, like GPU scaling and ModelMesh serving for high volume/density use cases. From the perspective of an eager engineer new to the KServe community, we will explore the KServe features that solve common issues for engineers and data scientists who are interested in or responsible for machine learning model deployment. Expect to learn about KServe’s fundamental offerings, like out-of-the box model serving and monitoring, and its exciting new, advanced functionalities, such as its inference graph capabilities and ModelMesh features. We will discuss the host of new features added to the project since its publication in 2019 and also outline KServe’s roadmap as it moves forward towards its v1.0 release.
A
Hi
everyone,
how
are
you?
My
name
is
alexa
griffith
and
I,
like
you,
said,
I'm
on
the
data
science
platform
team
at
bloomberg.
I've
been
there
now
for
a
few
months.
I
started
in
january
and
what
the
data
science
platform
team
does
is
we
support
multiple
machine
learning
features
from
turning
from
training
to
serving
models.
A
A
So
a
lot
of
things
are
new
to
me,
which
is
great,
and
I
find
that
I
enjoy
learning
new
things
by
you
know,
writing
or
discussing
those
topics
and
like
blogs
or
podcasts
and
sometimes
being
a
beginner
can
hopefully
bring
some
new
insight
or
new
perspective
into
a
topic,
and
I
like
to
have
fun
with
it.
So
they're
going
to
be
like
some
silly
sick
figure,
drawings
in
the
mix,
but
yeah.
So
that's
pretty
much.
A
What
I'm
doing
when
I
joined
bloomberg,
I
was
very
unfamiliar
with
k-serve,
so
I
find
that
sometimes
abstractions
on
top
of
kubernetes
can
be
a
little
confusing
or
a
little
a
little
interesting
to
talk
about.
So
in
this
talk
I'll
attempt
to
present
both
the
basic
and
advanced
features
of
k-serve.
From
that
perspective,
so,
as
you
can
tell,
this
is
my
first
pr
that
was
just
merged
for
k-serve,
I'm
a
really
big
contributor,
so
I'm
obviously
very
qualified
to
give
this
talk.
A
But
no,
this
is
a
joke,
but
this
is
my
first
of
hopefully
many
contributions
yeah.
So
let's
explore
k-serve.
So
does
anyone
know
who
lives
on
planet
k-serve?
Who,
oh
you,
don't
know?
Okay?
Well,
it's
the
k.
Natives
yeah,
apparently
there's
also
a
best
joke
a
competition
today
that
I
did
not
know
about,
but
I'm
not
saying
it
could
win,
but
it
might,
if
you
vote,
for
it,
anyways.
Okay,
so
given
the
environment
that
this
is
kubecon,
I'm
not
really
going
to
go
over
exactly
what
a
pod
or
what
a
container
is.
A
But
I
do
want
to
just
focus
on
the
basic
kubernetes
controller
pattern,
because
I
think
that's
relevant
to
my
talk
today.
So
kubernetes
implements
a
controller
pattern
and
it's
the
control
plane
is
in
the
dotted
line.
Each
one
has
some
components
and
each
component
is
in
charge
of
handling
different
parts
of
the
system.
So
the
api
server
is
like
the
front
end
handling
some
requests.
The
scheduler
manages
the
scheduling
and
eviction
of
the
nodes,
etc.
A
So,
similarly,
there
are
components
on
the
nodes
themselves
that
make
sure
everything's
running
smoothly
so
like
the
network
proxy
that
allows
communication
between
nodes
for
traffic
etc,
and
there
are
other
add-ons
that
you
can
plug
in
like
dns,
so
kubernetes
allows
you
to
create
these
things
called
custom
resource
definitions.
I'm
sure
most
of
you
are
familiar
with
this
and
custom
controllers.
So
when
you
combine
these
two,
you
get
a
declarative,
api
or
desired
state
system.
So
this
is
what
k-serve
does
it
extends
the
kubernetes
api.
A
A
Of
course
you
have
like
the
pipeline
development,
the
data
pipeline-
you
can
develop
it
and
there's
also
a
training
component
where
you
need
a
certain
data
set
to
train
the
model
and
then
once
you
train
the
model,
there's
also
an
inference
component,
so
you
use
the
model
in
some
live
machine
learning
algorithm
to
give
some
output.
So
today
I'm
going
to
focus
on
the
inference
component.
That's
what
caser
focuses
on
to
be
even
more
specific.
A
What
caser
focuses
on
what
I'm
going
to
call
core
inference
and
it
focuses
on
the
deploying
and
monitoring
of
the
core
inference
part
of
an
inference
service.
So
the
goal
of
case
serve
is
to
make
both
the
monitoring
and
deploying
for
inference
services
super
easy.
So
let's
discuss
the
components
that
make
up
part
of
the
machine,
learning
workflow
and
some
commonalities
between
them.
So
just
so,
we
know
what
we're
all
talking
about
before
we
get
into
it
into
what
k-ser
really
does
a
lot
of
inference?
Workflows
have
a
predictor
in
common.
A
So
what
exactly
is
a
predictor?
A
predictor
consists
of
a
model,
so
maybe
it's
something
like
a
tensorflow
run.
Some
regression
does
some
calculations
and
it
runs
on
a
serving
runtime
called
like
torch
serve
for
example.
So
all
it's
serving
a
model
server
is.
Is
it
wraps
the
model
instead
of
rest
api,
so
they
can
talk
to
the
model,
but
this
is
a
very
simple
use
case.
Usually,
the
interference
services
also
have
some
more
components
like
let's
say
what
we're
going
to
call
a
transformer.
A
So
a
transformer
is
optional,
but
very
commonly
used
and
it
costs
the
predictor
it
handles.
Pre
and
post-processing
of
the
data,
like
you
have
raw
data,
and
you
want
to
turn
it
into
vectors
before
you
before
you
hit
your
model,
so
you
can
also
plug
in,
and
this
is
a
very
simple
view
of
a
workflow
as
well.
A
You
can
have
a
lot
of
other
components,
so
you
plug
in
like
something
we
call
an
explainer
or
a
detector
to
detect
things
that
how
the
model
is
doing
and
we're
going
to
talk
about
those
a
little
later.
But
basically
this
is
a
transformer,
and
you
can't
just
make
this
and
go
to
production
right.
There
are
a
few
things
you
still
need
to
do:
to
have
a
production,
ready
and
front
service.
A
For
example,
you
might
not
be
able
to
use
any
of
the
out
of
the
box
model
servers
and
then
you
might
need
to
create
your
own,
so
that
takes
some
work.
Some
code
you're
going
to
need
an
ingress,
probably
to
send
requests
from
an
external
service.
You
need
metrics.
You
need
to
set
that
up
yourself
and
monitoring.
A
So
the
question
is
like:
how
can
this
be
made
easier?
Well,
income's,
k-serve,
so
k-serve
was
formerly
known
as
kf
serving
it
was
part
of
the
kubeflow
project
and
it's
taken
out
now
into
its
own
thing
and
it's
called
k-serve.
So
this
is
the
first
sentence
that
k-serve
has
on
its
github
and
I'll.
Just
read
it.
Real
quick
k-state
provides
a
kubernetes
custom
resource
definition
for
serving
machine
learning
models
on
arbitrary
frameworks.
A
It
aims
to
solve
production
model
serving
use
cases
by
providing
performant
high
abstraction
interfaces
for
common
ml
frameworks
like
tensorflow
xt
boost
scikit-learn,
pytorch
and
onyx
okay.
So
what
does
that
really
mean?
So?
Basically,
k-serve
is
a
custom
resource
definition.
It's
just
built
on
top
of
it's
built
on
top
of
and
abstracts
it's
underlying
layers
like
k-nav
and
seo,
for
example,
so
it
makes
it
really
easy
to
use.
You
have
a
very
simple
yaml
and
everything
is
created
underneath
for
you.
So
k-surf
supports
many
common
machine
learning
frameworks
that
data
scientists
use.
A
So,
let's
look
at
the
main
components
that
k-serve
is
built
upon.
So,
of
course,
on
the
lower
layer.
You
have
your
compute
cluster,
like
your
gpu
cpu
and
then
on
top
of
it,
we
have
kubernetes,
which
is
what
we're
running
on,
and
then
this
optional
layer
of
k-native
and
istio
k-nate
is
an
open
source
community
project.
It
adds
components
for
deploying
running
and
managing
services,
serverless
applications
and
kubernetes,
if
you're
not
familiar
with
it.
A
A
Okay,
so
here
are
some
of
the
main
features
of
case
serve.
It
allows
you
to
scale
up
and
down
from
zero
to
and
from
zero.
You
can
have
request
space,
auto
scaling
based
on
your
cpu
or
gpu.
It
also
offers
request
batching.
If
that's
what
you
want
and
then
you
can
do
requests
and
response
logging
as
well.
A
It
offers
security
with
authentication
and
authorization,
traffic
management
distributed
tracing
and
out-of-the-box
metrics.
So
here's
a
few
of
them
and
it
also
offers
like
rollout
strategies
like
canary
rollout,
which
I'll
do
a
little
demo
on
later,
but
here
are
just
a
few
of
the
main
features
of
k-serve
and
next,
let's
talk
about
how
we're
actually
able
to
achieve
that.
A
Okay.
So
let's
talk
about
the
predictor,
we
talked
about
a
predictor
before
like
in
general
terms,
but
let's
walk
through
how
k-serve
enhances
the
single
model
use
case
of
a
predictor.
So
first
we
have
something
called
a
storage
initializer,
which
is
an
init
container.
So
it
comes
up.
It
reaches
out
to
your
model
storage.
Wherever
your
model
is
stored,
there's
multiple
options
in
case
there
for
that
download
your
model
and
then
go
away.
So
the
nexus,
the
other
two
things
that
are
going
to
be
up
running
are
the
q
proxy.
A
This
is
a
k
native
component,
so
it
is
optional,
but
it
routes
the
request
to
a
k-serve
container
with
max
concurrent
requests
configured.
So
it
gives
you
some
rate
limiting
and
the
main
part
of
the
inference
service
is
the
model
and
the
model
server
like
we
talked
about,
and
it
allows
you
to
easily
plug
in
some
things
out
of
the
box,
as
well
as
create
your
own
custom
model
server.
A
So
if
you
can
remember
from
the
control
plane
slide,
the
control
plane
handles
the
state
of
of
the
system.
So
the
inference
service
defines
the
state
and
the
k-12
k-serve
controller
is
in
charge
of
reconciling
that,
so
we
make
sure
we're
getting
to
the
desired
state.
It
creates
all
the
resources
for
the
infant
service
needs
below
it.
A
So
this
is
just
a
higher
level
overview
of
what
we
just
went
through
again.
You'll
see
the
model
storage
there.
Like
I
mentioned,
there
are
multiple
of
types
of
model
storage.
That's
supported
through
k-serve,
like
s3,
uri,
azure
and
persistent
volume
claims
and
more
being
added
in
this
next
version,
as
well,
so
more
options
there.
A
I'm
not
going
to
go
into
each
of
these,
but
I
think
this
is
just
helpful,
especially
as
someone
who's
learning
about
this
to
get
a
view
of
all
the
resources
that
are
created
and
then
you
can
like
cut
or
describe
or
look
into
each
of
those
resources
and
like
see
see
what
their
settings
are.
So
just
for
understanding
how
k-serve
works.
I
thought
this
was
super
helpful.
A
Okay,
so
next
now
that
we
talked
about
the
control
plane,
let's
talk
a
little
bit
about
the
data
plane.
So
let's
follow
a
request
path.
So
you
have
your
ai
application.
It
hits
a
transformer
service.
Then
it's
going
sorry.
Then
it's
gonna
hit
the
transformer
pod.
Okay,
there
we
go.
It's
gonna
hit
the
transformer.
A
It's
gonna,
hit
the
q
proxy
and
hit
the
transformer
prod
and
it's
gonna
maybe
do
some
pre-processing
and
then
it's
gonna
reach
out
to
the
predictor
service
and
then
it's
going
to
hit
hit
the
predictor
pod
and
that
q
proxy
and
then
it's
going
to
run
the
model.
It's
going
to
run
your
inference
for
you,
okay,
so
that
was
the
david
plane
and
now,
let's
talk
about
something,
that's
interesting!
A
That's
just
added
the
inference
graph,
so
the
inference
services
can
be
set
up
as
a
chain
pipeline
with
k-serve,
which
is
super
nice
and
useful,
and
this
new
feature
allows
that
this
new
feature
allows
for
chaining
them
in
different
ways
and
the
case
their
contributors
have
been
working
very
hard
on
it.
So
I
just
think
some
of
the
different
modes
you
can
use,
so
you
can
use
sequence
node,
which
allows
you
to
connect
two
inference
services
in
a
sequence
which
is
like
how
I
have
it
there.
A
You
can
also
use
this
mode
called
switch
node,
which
allows
users
to
select
an
inference
service
to
handle
requests
based
on
a
condition
so
kind
of
like
which
one
do
you
want
to
go
to.
Splitter
node
allows
users
to
split
traffic
between
multiple
inference
services
using
a
weighted
distribution,
and
you
can
also
use
something
called
ensemble
node,
which
allows
users
to
route
requests
to
multiple
models
in
parallel.
A
So
another
important
feature
of
k-serve
is
that
it
uses
a
standard
standardized
inference
protocol
across
all
of
the
serving
run
times
so
that
people
can
easily
switch
from
say
tensorflow
to
pytorch
without
needing
to
change
the
way
you
talk
to
the
inference
service,
so
this
also
makes
it
really
easy
for
testing
and
benchmarking
for
the
user.
So
it's
just
something:
that's
really
nice
for
users
to
be
able
to
use
and
be
able
to
switch
between
models
that
they
need
to.
So
I
have
a
cute
little
animation
here,
he's
like
hi.
A
A
So
we
can
do
something
called
explain.
So,
instead
of
hitting
the
prediction
point
and
this
one
you
would
hit
explain
which
would
then
hit
predict
and
there's
multiple
explainers
you
can
use
so
basically,
the
way
I
understand
is
that
explainer
says:
why
did
my
model
make
this
prediction
for
a
given
instance?
So
if
you
have
an
image
and
it
defines
that
image
as
a
dog,
then
then
it
will
explain
what
part
of
the
image
is
most
relevant
to
making
that
result.
A
You
can
set
things
like
max,
latency
and
max
bat
batch
size
and
then
also
there's
a
monitor
and
logger
plugin
that
you
can
add,
so
it
has
asynchronous
payload
logging
monitoring
the
distributed,
the
distribution
of
incoming
requests
with
detectors
and
we'll
talk
about
a
few
of
those
detectors
on
the
next
slide.
You
can
also
hook
up,
alerting
and
also
I
see,
people
taking
pictures
which
is
totally
fine,
but
just
let
you
know
I
do
have
these
uploaded
as
well.
A
So
you
can
have
good
observable
good
observability
into
your
inference
service,
yeah,
okay,
so
we
discussed
a
little
bit
about
some
model
serving
run
times,
but
there
are
a
few
of
them
and
caser
keeps
adding
support
for
more
out
of
the
box.
For
example,
you
can
use
the
triton
inference,
server
tort,
serve
sklearn
sc
boost
just
some
of
the
few
ones
you
can
use
and
you
can
use
with
those
tensorflows
ports,
torch
script,
pytorch,
onyx
and
xgboost.
A
So
how
does
this
actually
look?
You
can
define
the
serving
runtime
if
you
want
and
the
or
you
can
divide
the
model
format
in
the
yaml.
So
this
is
like
a
very
basic
inference
service.
This
is
what
the
yama
would
look
like,
and
you
can
define
that
you're
using
say,
sk,
an
sk
learn
model
and
from
here
in
the
new
format
and
the
v2
format,
k
server
will
be
able
to
just
infer
which
wallet
serving
runtime
you're
running.
You
don't
even
have
to
define
it,
but
if
you
want
to
you
can
so
what
does
this?
A
A
A
What
happens
if
you
have,
I
don't
know,
let's
say
like
five
models,
so
you
at
least
have
tin
pots
running,
but
you
also
probably
have
a
transformer
and
for
each
replica
you
probably
have
another
one,
so
you
probably
at
least
have
for
each
model.
Four
pods
running
so,
as
you
can
tell,
this
is
okay,
for
maybe
you
know
five
models.
What
happens
when
you
get
to
like
a
hundred
or
a
thousand
models?
Obviously
this
isn't
very
scalable
and
it's
very
costly.
So
kubernetes
also
has
this
max
pod
limit
about
per
cluster.
A
A
So
there's
one
way
we
can
do
better
and
that's
called
multi-model
serving
so
multi-minus
serving
is
designed
to
address
three
types
of
limitations
that
case
server
run
into
that
we
just
discussed
so
the
compute
resource
limitation.
There's
an
overhead
of
running.
You
know
just
one
model
per
pod:
the
maximum
pod
limitation
and
the
maximum
ip
address
limitation.
A
A
Each
pod
can
only
have
so
many
models
and
those
have
to
be
defined,
so
this,
in
this
case,
you'll
still
run
into
some
limitations.
If
you
start
running
at
scale-
and
you
have
a
lot
of
models,
don't
worry,
though,
because
the
caser
contributors
have
been
working
very
hard
to
find
an
even
better
solution
for
a
high
volume,
high
density
use
case,
and
we
call
this
multi-model
serving
with
model
mesh.
A
Okay,
so
imagine
that
you
have
a
large
number
of
models
deployed
which
isn't
uncommon.
So
an
example
of
this
use
case
would
be
like
a
news
classification
service.
You
may
have
a
model
for
every
type
of
news
category
and
for
privacy
and
security
reasons
you
might
need
to
train
those
at
the
user
level.
A
So
a
lot
of
small
running
models,
so
using
the
many
models
in
this
use
case
will
have
the
benefit
of
a
better
and
more
accurate
results,
but
you're
going
to
run
into
a
lot
of
the
issues
we
just
discussed
with
inference
services
at
this
scale
running
on
a
kubernetes
cluster.
So
another
use
case
say:
let's,
let's
say
like
neural
network
based
models
are
best
suited
for
gpus,
but
gpus
are
usually
expensive
and
running.
A
Many
of
these
is
costly,
so
using
something
like
what
we're
going
to
describe
here
in
model
mesh
is
usually
a
good
option.
So
multi-serving
multi-model,
serving
with
model
mess,
is
a
solution.
This
problem
and
it
allows
infant
services
to
scale
so
it
decreases
the
average
resource
overhead
per
model,
so
it
becomes
more
cost
efficient
and
the
number
of
models
that
can
be
deployed
in
a
cluster
are
no
longer
limited
by
the
maximum
power
limitation
and
ip
address
limitation.
So
what
does
this
do?
A
Basically,
you
have
the
mesh
which
is
routing
the
routing,
the
request
you
have
the
puller,
which
is
is
replaced
by
or
replaces
the
storage
initializer
in
a
way.
So
in
storage
initializer
we
had
a
knit
container
that
came
up,
pulled
the
model
and
then
left
now
we
have
a
puller
which
is
a
long
running
side
car
which
is
constantly
pulling
models
as
needed.
A
So
that's
a
very
simple
use
case,
of
course,
you're
probably
going
to
have
again
like
at
least
one
replica
and-
and
this
just
shows
that
you
can
have
multiple
models
on
different
pods
running
or
different
replicas
of
the
same
serving
run
time
so
molar
server
x
and
the
left
plot
can
have
model
a
b
or
c
and
then
the
right
part.
It
can
have
any
other
number
of
models
that
run
on
model
server
x.
A
Okay,
so
now
I
have
a
small
or
a
very
short
demo.
Let's
see
this
is
just
going
to
be
setting
up
an
inference
service.
This
is
like
this
is
one
of
our
okay.
This
is
one
of
our
main
examples
about
how
to
set
up
an
inference
service
so
yeah
I
made
a
I
made
a
namespace
called
case,
serve
contacts,
and
then
I
just
set
up
one
of
the
the
sk
learn
yammels
for
an
inference
service,
which
is
very
similar
to
what
I
just
showed
you.
A
I
already
had
it
applied
because
I
didn't
want
like
any
issues
or
I
have
to
wait
for
it
to
come
up,
and
I
show
that
it's
already
running-
and
here
I'm
just
like
looking
at
different
resources.
So
this
is
the
inference
service
resource
you
can
see.
It's
ready,
true
and
latest
is
like
100
of
the
traffic
is
going
there,
I'm
setting
up
the
ingress
gateway,
because
I
want
to
port
forward
and
send
send
a
request
to
the
predictor
to
get
some
prediction
results.
A
Okay,
so
I
put
forward
there
and
I
set
my
ingress
port-
here's
my
input,
it's
just
a
very
simple
input
that
I'm
going
to
that
I'm
going
to
submit
into
the
inference
service,
and
now
I
curl
it
and
you
can
see
under
predictions.
We
have
an
answer.
Okay.
So
what
I'm
going
to
show
next
is
a
quick
example
of
using
a
canary
rollout,
which
is
one
of
the
features
of
case
serve
so
here
I'm
going
to
add
a
canary
traffic
percent
at
10.
A
A
So
here
I'm
checking
you
can
see
now
it's
ready
with
90
on
the
previous
version
and
10
of
traffic
on
the
latest
version,
and
then,
if
I
remove
that,
I
can
promote
this
version
to
okay.
I
can
promote
this
version
to
100
traffic
and
because
of
time,
oh
yeah,
okay,
now
it's
good
yay,
okay,
there's
a
little
demo
and
really
quickly.
There
are
a
lot
of
cool
new
features
coming
out.
I'm
just
going
to
leave
this
up
because
we
don't
have
a
lot
of
time
but
be
sure
to
go.
A
D
A
C
A
E
Hi
thanks
for
the
talk
by
the
way,
the
I
have
a
question
about
multi
model
use
case
that
you
talk
about
the
mesh.
So
what,
in
the
in
this
case,
whenever
a
request
comes
in,
is
it
at
run
time?
The
model
is
pulled
or
like
just
at
the
moment
when
the
request
comes
in
or
how
does
it
work?
Because
I
see
that,
okay,
that
there
are
models,
are
you
said
that
the
initializer
is
replaced
with
something
called
puller
or
something
like
that?
A
Yeah
so
in
model
mesh,
the
question
was
about
the
differences
in
the
storage
initializer
and
the
puller
so
and
the
model
and
the
model
mesh.
The
storage
initializer
is
replaced
by
the
puller,
which
is
a
long
running
sidecar.
So
it's
able
to
pull
models
as
needed
as
you're
as
you're,
hitting
that
that
model
yeah.
That's.
B
B
A
D
A
Definitely
so
scale
to
zero
is
something
you
can
configure.
Sometimes
you
want
it
to
save
resources,
but
for
other
mission,
critical
things
you
may
not
ever
or
things
that
can't
handle
the
latency
of
a
pod
starting
up
when
it's
been
down,
but
for
those
use
cases
you
wouldn't
want
to
use
scala
zero
or
something
that
used
to
be
up
all
the
time.
Yeah.