►
Description
Don’t miss out! Join us at our next event: KubeCon + CloudNativeCon Europe 2022 in Valencia, Spain from May 17-20. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Serving Machine Learning Models at Scale Using KServe - Yuzhui Liu, Bloomberg
KServe (previously known as KFServing) is a serverless open source solution to serve machine learning models. With machine learning becoming more widely adopted in organizations, the trend is to deploy larger numbers of models. Plus, there is an increasing need to serve models using GPUs. As GPUs are expensive, engineers are seeking ways to serve multiple models with one GPU. The KServe community designed a Multi-Model Serving solution to scale the number of models that can be served in a Kubernetes cluster. By sharing the serving container that is enabled to host multiple models, Multi-Model Serving addresses three limitations that the current ‘one model, one service’ paradigm encounters: 1) Compute resources (including the cost for public cloud), 2) Maximum number of pods, 3) Maximum number of IP addresses. 4) Maximum number of services This talk will present the design of Multi-Model Serving, describe how to use it to serve models for different frameworks, and share benchmark stats that demonstrate its scalability.
A
A
A
Those
functionalities
include
how
to
do
data
exploring
using
jupyter
notebook
how
to
train
models
using
popular
frameworks
such
as
tensorflow
pytorch,
escalar
extra.
We
also
manage
experiments
and
do
online
inference.
Today,
I'm
going
to
talk
about
the
open
source
project
we
initiated
together
with
multiple
collaborators
named
kaiser.
A
We
also
run
into
recent
problems.
We
want
to
deploy
many
models
at
scale,
and
I
want
to
talk
about
how
we
address
this
scalability
problem
and
first,
I
want
to
give
you
a
little
bit
background
about
k.
Serve
k
serve
is
previously
known
as
cave
serving
if
you
are
already
familiar
with
scale
serving
if
you
are
already
deploying
cave
serving
into
your
platform.
Caser
is
the
same
as
cave
serving.
A
K-Surf
has
experienced
multiple
important
milestones
in
september
2019.
We
released
the
first
version
of
k-serve
as
a
sub
project
under
coop
flow.
It
was
released
under
the
name
kf
serving
a
few
months
later.
We
introduced
it
at
coupon
us
and
we
spent
the
next
year
and
a
half
to
develop
this
project
and
we
released
its
stable
version,
v1
beta
1
version
into
2021..
A
A
Here
are
linked
to
two
important
articles
and
blog
posts.
You
are
very
welcome
to
follow
the
links
to
learn
over
announcement
together
with
coop
flow
community
and
also
the
blog.
We
posted
tech
at
bloomberg
that
talk
about
our
journey
of
building
a
production
grade.
Machine
learning
model
sewing
solution.
A
For
those
of
you
who
are
less
familiar
with
inference,
this
diagram
demonstrates
a
typical
and
also
simplified
model
development
life
cycle.
Normally,
overuser
are
data
scientists
and
machine
learning
engineers.
They
will
first
prepare
data
and
use
a
machine
learning
framework
to
train
a
model,
and
once
the
model
is
trained,
our
user
wants
to
deploy
this
model
into
a
production
system,
and
this
model
should
be
able
to
answer
real-time
questions
such
as,
given
two
sentences
can
you
can
my
model?
Tell
me
what
is
the
similarities
between
these
two
sentences
or
given
a
news
article?
A
A
First,
we
want
to
think
about
the
cost
of
deploying
such
a
machine
learning
model,
how
much
cpu,
how
much
memory,
how
much
gpu
resource
is
needed
and
when
there's
no
request
coming
into
the
service,
is
there
a
way
to
automatically
scale
it
up
or
scale
it
down?
We
also
want
to
monitor
our
machine
learning
services.
A
A
A
A
A
A
A
So
here
comes
caser,
so
k-serve
is
a
highly
scalable
and
standard-based
model
inference
platform
on
kubernetes
for
trusted
ai
at
the
lowest
level,
is
our
computer
resource.
In
normally
in
a
kubernetes
cluster,
we
have
a
collection
of
cpu,
gpu
and
memory,
sometimes
even
tpu.
Those
are
the
resources
we
have
for
computing.
A
On
top
of
all
the
computing
resources,
we
run
a
kubernetes
layer.
Kubernetes
is
used
as
a
way
to
orchestrate
and
manage
all
the
compute
resource
on
top
of
the
kubernetes
layer.
This
is
over
serverless
layer.
We
use
k-native
and
ecl
to
build
the
serverless
layer.
With
this
layer
we
are
able
to
automatically
scale
up
and
scale
down
according
to
the
incoming
traffic.
A
A
A
A
The
model
server
can
handle
incoming
requests
and
around
inference
result.
The
second
important
control
component
is
transformer.
Sometimes
users
want
to
implement
customized,
pre-process
and
post-process
steps
to
translate
data
point
into
a
format
that
the
predictor
can
understand
and
then
post
process
it
back
to
a
format.
That's
an
application
called
understand.
So
that's
how
the
customized
implementation
can
be
integrated
into
k-serve.
A
A
A
very
critical
part
of
k
surf
is
we
define
a
standard
inference
protocol
this
standard
we
work
very
closely
with
multiple
model
server
community,
including
triton,
torch
surf
and
ml
server.
We
make
sure
we,
the
case
of
community,
can
come
up
with
a
set
of
consistent
inference
protocol
to
provide
a
unified
user
experience.
A
A
A
A
So,
let's
take
a
look,
how
the
current
approach
works.
So
how
does
how
currently
k
serve?
Deploy
a
model,
a
machine
learning
model?
The
gray
box
here
represents
inference
platform
cluster
and
in
this
cluster,
each
of
our
users
on
their
own
name,
space,
which
is
represented
by
the
blue
box,
the
light
blue
box
and
in
their
own
name
space.
They
can
run
multiple
inference
services
and
this
inference
service
will
fetch
a
model
from
external
model
storage.
A
A
So
what
kind
of
problem
this
approach
bring
us
when
we
want
to
scale
up
number
of
models,
because
when,
if
we
want
to
scale
up
number
of
models,
we
essentially
need
to
scale
up
the
number
of
services
we
run
in
our
platform,
which
doesn't
scale
very
well,
and
I
will
talk
about
the
scalability
limitations
we
foresee
when
we
want
to.
If
each
team
want
to
run
hundreds
even
thousand
models
in
our
inference
platform.
A
A
A
It
can
also
do
certain
batching
and
logging,
so
those
sidecar
has
a
needs,
certain
amount
of
cpu
and
memory
to
run
alongside
the
model
server.
So
let's
that's
the
config,
the
resource
the
sidecar
requires
is
configurable,
but
let's,
for
example,
let's
think
each
side
car
takes
0.5,
cpu
and
0.5
gigabyte
memory
overhead.
Based
on
this
configuration.
A
The
second
limitation
we
foresee
that
we
are
going
to
run
into
is
the
maximum
pod
limitations.
Some
of
you
may
be
already
familiar
with
the
kubernetes
default
setting
on
each
node.
By
default,
we
can
run
110
parts
and
based
on
the
official
documentation
from
kubernetes
scalability
best
practice.
We
shouldn't
really
run
more
than
100
parts
per
node.
A
A
The
number
of
ip
address
available
in
each
kubernetes
cluster
varies
a
lot
depending
on
how
the
admin
manages
this
cluster.
But
I
want
to
point
out
that
this
is
a
in
one
of
the
testing
cluster.
We
run
a
test,
we
have
several
thousand
ip
address
available
and,
based
on
that
limitation,
we
can
run
like
several
thousand
models.
A
This
is
the
diagram
of
model
mesh
solution,
so
let
me
walk
you
through
it
from
the
top
to
the
bottom
at
the
top.
It
is
the
machine
learning
application
which
sends
inference,
requests
into
model
mesh
and
one
model
mesh
can
contain
multiple
serving
runtime
different
serving
runtime
is
essentially
a
different
type
of
model
server.
In
this
diagram,
we
there
are
two
serving
runtime
available.
Different
serving
runtime
can
produce
service
solutions
for
different
type
of
machine
learning
model.
A
A
A
The
mesh
sidecar
also
acts
as
a
router.
The
model
mesh
store
model
to
part
it
routing
table
in
the
lcd
in
external
fcd.
So
when
there's
inference,
requests
coming
in
the
sidecar
will
look
up
the
routing
table
and
it
will
figure
out
which
model
is
loaded
into
which
pile
id
and
routes
the
request
to
the
correct
pot.
According
to
the
routing
table.
A
And
so
you
now
may
be
curious
to
learn
what
kind
of
service
runtime
model
mesh
can
provide.
So
out
of
box
integration
we
have,
we
will
provide
triton
inference
server,
which
is
developed
by
nvidia's,
which
is
developed
by
nvidia.
This
model
server
can
serve
machine
learning,
framework
models
trained
by
emotional
frameworks
such
as
tensorflow,
pytorch,
tensor,
rt
or
onyx.
We
also
by
default,
integrate
with
a
sounds
ml
server.
This
zelda's
ml
server
is
a
python
based
server.
It
can
serve
frameworks
such
as
sklen
xgboost
or
like
gbm.
A
So
a
lot
of
you
may
be
very
curious
to
know
what
kind
of
performance
of
model
match
can
provide
if
we
co-locate
multiple
models
into
the
same
part.
Will
that
have
impact
on
the
latency
of
throughput.
This
solution
can
provide
so
we
did
a
performance
test.
This
performance
test
was
done
on
a
single
node,
8
cpu,
64
gigabyte,
ram
cluster,
and
we
deployed
a
very,
very
simple
string
model.
It's
all.
It
only
has
around
700
bytes.
A
This
latency
test
is
done
by
running
two
triton
serving
run
times,
and
we
gradually
increase
the
qps
from
25
per
second
all
the
way
to
2800
per
second.
Each
performance
test
will
run
for
60
seconds
for
each
qps.
We
also
gradually
increase
the
density
of
the
model
mesh
from
1000
2000,
all
the
way
to
20
20
000..
A
A
A
A
A
A
A
B
A
B
A
Okay,
so
thank
you
very
much
for
the
question,
so
the
question
is
that
I
mentioned
that
there's
a
compute
resource
overhead
come
with
each
inference
service.
The
question
is
that
if
we
start
to
use
gpu,
what's
the
change,
what
kind
of
change
it
is
about
the
overhead,
so
there's
nothing
changed
so
in
this,
for
example,
if
we
take
a
look
at
the
diagram,
the
model
one
will
be
deployed
into
a
model
server,
so
model
server
is
the
one
that
requires
cpu
or
gpu
for
inference.
A
So
that's
one
independent
container
around
inside
the
inference
service
and
alongside
that
model
server,
there's
another
container.
That's
the
extra
container
running
as
a
side.
Car
requires
extra
cpu
and
memory.
So
changing
the
model.
Server
from
using
cpu
to
gpu
doesn't
really
change
the
side.
Cars
requirement
for
computer
resource.
A
That's
a
very
good
question,
so
if
some
of
you,
this
may
lead
to
some
discussion
about
slicing
gpu
so
currently,
if
we
want
to
request
gpu
resource
in
kubernetes,
there's
no
very
straight
straightforward
or
easy
way
to
request
a
slice
of
a
gpu.
So
I
know
in
the
managed
compute
in
the
mesh
kubernetes
concept,
it's
very
easy
to
think
about.
Well,
I
have
a
gpu
allocate
to
this
inference
service.
Can
I
request
a
slice
of
the
gpu
and
just
to
use
that
for
the
sidecar?
A
The
answer
is
that
there's
no
very
easy
way
to
do
that.
So
when
we
request
gpu,
we
request
a
full
gpu
actually
in
real
use
cases.
Very
often
we
notice
that,
even
though
a
model
server
requests
a
full
gpu,
but
when
the
inference
happens,
it
won't
really
use
the
full
capacity
of
the
gpu.
Sometimes
it
doesn't
even
use
the
full
capacity
of
the
memory.
So
if
we
take
a
look
at
typical
gpu,
most
of
them
come
with
32
gigabyte
memory,
so
some
models
actually
really
small.
A
A
The
amount
of
computing
power
you
consume
heavily
depend
on
the
volume
of
requests
that
you
send
into
the
model
server.
So
when
you
have
a
lot
of
concurrent
requests
sent
to
the
to
the
model
server,
that
only
has
a
gpu
allocated
to
it.
You
can
notice
that
the
gpu
consumption
world
goes
up,
but
during
the
time
that
there's
a
very
small
amount
of
volume
coming
to
the
gpu
powered
model
server
like
you,
it's
very
obvious
that
the
consumption
of
gpu
computing
power
decrease
a
lot.
A
A
D
But
isn't
it
a
little
bit
against
the
kubernetes
model?
Reason?
I'm
saying
that
one
thing
is
all
of
these
model
competing
for
the
resources,
because
you
don't
never
don't
have
isolation
of
resources,
so
there
is
no
quality
of
service.
One
model
is
taking
more
computer
resources.
It
will
impact
rest
of
the
models.
Second
thing:
if
this
part
goes
down
or
this
node
goes
down,
all
the
model
goes
down
together.
A
Yeah,
that's
a
very
good
question.
Each
model
can
take
some
different,
a
different
amount
of
computing
service.
Actually,
that's
a
we
internally
over
working
group.
We
had
a
long
discussion,
so
the
original
design
of
the
solution
is
that
each
model
of
let
me
go
back
to
the
to
this
diagram.
One
of
the
original
idea
we
have
is
that
each
model
will
have
the
same
amount
of
replicas
that
just
spread
among
all
the
replicas
belongs
to
a
model
server,
and
then
we
start
to
realize.
A
Let's
say
we
have
model
a
and
b
and
c
in
the
same
part
and
the
model
a
takes
most
of
the
request
and
a
b
and
c
only
take
like
one
request
per
hour,
but
because
of
model
a
need
to
handle
really
high
volume,
so
model
a
will,
drive
the
same,
serving
runtime
cost
up
into
like
many
replicas
like
10
20,
but
motor
b
and
c
will
be
forced
to
scale
up
together.
So
that's
that's
the
idea
like
we
can.
A
We
spend
a
lot
of
time
discussing
and
that's
why
we
moved
to
the
model
mesh
idea,
so
we
can
scale
up
different
model
differently.
So,
let's
think
of
the
diagram
again,
the
model
b,
it
has
more
replicas,
so
essentially,
collectively
model
b
owns
more
computing
resource
in
this
serving
runtime
and
model.
A
only
has
one
replica
so,
like
collectively
model
a
only
owns
like
less
computing
resource.