►
From YouTube: Introducing KFServing: Serverless Model Serving on Kubernetes - Ellis Bigelow & Dan Sun
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Introducing KFServing: Serverless Model Serving on Kubernetes - Ellis Bigelow, Google & Dan Sun, Bloomberg
Production-grade serving of ML models is a challenging task for data scientists. In this talk, we'll discuss how KFServing powers some real-world examples of inference in production at Bloomberg, which supports the business domains of NLP, computer vision, and time-series analysis. KFServing (https://github.com/kubeflow/kfserving) provides a Kubernetes CRD for serving ML models on arbitrary frameworks. It aims to solve 80% of model serving use cases by providing performant, high abstraction interfaces for common ML frameworks. It provides a consistent and richly featured abstraction that supports bleeding-edge serving features like CPU/GPU auto-scaling, scale to and from 0, and canary rollouts. KFServing's charter includes a rich roadmap to fulfill a complete story for mission critical ML, including inference graphs, model explainability, outlier detection, and payload logging.
https://sched.co/UaZo
A
B
B
B
Cool,
so
chaos,
Irving
is
a
new
project
in
the
coop
flow
ecosystem.
Ku
flow.
You
probably
are
aware
of
right
now,
if
you're
coming
to
the
ml
track
at
coop
con,
but
if
you're
not
coop,
grows
or
could
flow
as
a
project
in
in
the
kubernetes
ecosystem
for
training,
managing
deploying
models
on
kubernetes
and
KF
serving
is
the
serving
story
for
coop
flow.
B
So
if
you
have
a
model,
what
is
a
model?
Typically,
a
data
data
scientist
takes
a
bunch
of
data
that
they
have
and
they
write
some
model
code.
Maybe
that's
tensorflow
or
pi/2,
or
XG
boost
they're
different
frameworks.
Sometimes
it's
just
some
raw
Python
code
and
you
take
you,
take
your
algorithm,
your
model
and
you
train
it
against
some
data
and
you
eventually
end
up
you're,
so
vast
oversimplification,
but
eventually
end
up
with
a
some
sort
of
asset
that
can
be
served
for
tensorflow.
This
is
a
serialized
asset
called
a
SAV
model.
B
Proto
different
frameworks
have
different
serialization
schemas,
but
as
a
data
scientist,
you
end
up
with
this
resource
that
you
then
need
to
figure
out.
Okay,
how
do
I
start
sending
traffic
to
this
thing
and
some
data
scientists
have
started
to
come
to
kubernetes
and
say:
okay.
Well,
kubernetes
is
good
at
surveying
and
I
need
to
serve.
How
do
I
do
that,
and
this
is
what
they
need
to
learn.
B
B
So
this
is
your
domain.
You
understand
it
kind
of
well,
but
for
data
scientists,
it's
very,
very
terrifying,
and
it's
not
even
their
job
right.
They
need
to
focus
on
new
algorithms,
new
frameworks,
reading
papers
going
to
ml
conferences,
not
infrastructure
conferences.
These
fields
are
also
both
changing
incredibly
quickly,
which
is
really
really
challenging
to
stay
on
top
of
both.
So
this
is
where
KF
serving
it
comes
in.
We
try
to
take
all
of
the
complexity
around
infrastructure
and
package
it
up
into
a
very,
very
simple
consumable
interface
for
data
scientists,
data
scientists,
our
customer.
B
That
means
we're
solving
problems
for
them,
not
infrastructure,
people,
it's
an
infrastructure
project
and
so
infrastructure
is
always
involved.
There's
some
handoff
without
some
relationship
there,
but
as
we
build
our
interface
and
as
we
build,
our
features
were
focused
on
meeting
data
scientists
where
they
are
we're
trying
to
build
a
consistent
experience
for
deploying
models
of
any
kind,
whether
it's
PI,
torch
or
XT
boost.
We
don't
care,
we
don't
have
any
opinions.
We
want
to
support
all
of
the
popular
frameworks.
We
want
a
really
simple
use
case,
but
we
want
it
to
be
extensible.
B
As
well,
that
means
on
day
100
or
a
year
in
once
your
model
is
mission-critical,
you
don't
we
don't
want
to
have
our
users
have
that
moment
of?
Oh
no,
how
do
I
add
Network
policy
that
should
just
work?
We
also
don't
think
that
modern
production
features
that
you've
kind
of
see
with
kubernetes
like
canary
deployments,
tracing,
etc.
We
don't
think
that
ml
workloads
shouldn't
have
those
features,
because
ml
is
so
complicated.
B
I
then,
a
couple
weeks
later
got
introduced
to
Dan
son,
who
was
doing
the
exact
same
thing
at
Bloomberg
and
Microsoft
was
doing
a
similar
thing
and
I
thought.
Okay,
god,
we've
got
a
stop.
Let's
talk
about
what
we're
doing
talk
about
what
we're
trying
to
achieve
and
see
if
we
can
work
together
on
this.
All
of
these
projects
were
an
open
source,
but
they
weren't
really
compatible
with
each
other.
B
We
weren't
really
paying
attention
to
each
other,
so
we
founded
this
working
group
that
we
called
KF
serving
and
we
started
to
develop
an
interface
and
a
system
that
met
all
of
our
customers
needs.
So
we
have
a
huge,
a
diverse
set
of
customers
at
every
set
of
companies
and
we're
trying
to
solve
the
model
serving
problem
once
in
a
way.
That's
general
COO
flow
was
a
really
awesome
community
to
help
us
build.
This
coop
flow
has
some
really
amazing
principles
around
open
governance
and
open
community.
B
It
was
really
easy
for
people
to
trust
coop
flow
as
a
standard
and
is
a
good
place
to
go,
develop
ml
on
kubernetes.
It
was
it
wasn't
even
that
hard
of
a
self
people
just
started
showing
up
to
our
weekly
meetings,
and
before
long
we
had
a
technical
steering
committee
from
five
different
companies,
the
ones
you
see
here
and
we
were
having
weekly
discussions
on
how
to
develop
AF
serving
we
started
off
with
an
interface.
This
is
I
think
one
of
the
most
important
pieces
to
making
that
day.
One
experience
for
data
scientists
really
easy.
B
Here's
three
examples
of
our
CRD
one
for
scikit-learn,
one
for
one
for
tensorflow
and
one
for
pi
torch.
You
can
see
that
this
will
get
you
three
different
applications,
each
with
an
HTTP
endpoint
and
each
one
serving
a
model
of
a
different
framework
in
each
of
these
examples.
There's
a
storage,
your
eye
that
points
somewhere
in
the
cloud
in
this
case
Google
Cloud,
Storage
to
a
serialized
asset,
one
for
SK
learn
one
for
tensorflow
and
one
for
pi
torch.
B
B
We
have
these
concepts
of
framework
specs,
so
there's
a
tensor
flow
spec,
a
PI,
torch,
spec
and
they're
common
enough,
in
that
they
have
the
same
specification
for
storage,
your
eye,
things
like
kubernetes
resource
requests,
etc,
but
any
extra
fields
that
they
need,
for
example,
tensorflow,
batching
or
pi
torch
class
name.
We
also
wire
those
through
the
spec.
B
If
we
dig
a
little
bit
deeper
into
this
example,
and
in
here
you
have
kind
of
the
day,
one
I
just
want
an
endpoint
that
serves,
and
here
you
want
to
actually
start
to
exhibit
some
control
over
it.
We
start
off
with
the
the
first
extension,
which
is
a
service
account.
Potentially
you
want
to
authenticate
your
request
to
Google
Cloud
Storage,
because
you
don't
want
it
to
be
public
and
you
need
an
identity
running
on
the
container.
B
We
allow
you
to
pass
in
a
kubernetes
identity
that
we
mount
on
the
container
and
it
just
wires
up
magically
min
and
max.
Replicas
are
a
way
to
control
your
provisioning
to
make
sure
that
you
don't
drop
requests
and
that
you
can
meet
demand,
while
not
overspending
and
resource
requests
are
obviously
very
important.
We
have
some
defaults,
but
most
of
the
time
you
want
to
customize
this
for
your
model.
We
even
allowed
GPUs
through
this
interface.
B
The
second
half
of
the
spec
is
a
somewhat
controversial
but
I
think
really
powerful
piece
of
KF
serving.
If
you
think
about
how
to
do
canary-
and
this
is
our
canary
mechanism-
how
to
do
canary
as
an
ops
engineer.
Typically,
you
would
create
multiple
kubernetes
deployments,
maybe
set
up
a
nist,
EO
virtual
service
and
route
between
them.
Other
systems
like
k,
native,
allow
you
a
separate
interface,
that's
fairly
similar
to
do
traffic
splitting
and
canary
deployments,
but
that
can
be
really
challenging,
especially
for
data
scientists.
B
We've
we
basically
simplified
the
n
way
traffic
split
into
a
two
way:
traffic
split,
with
the
assumption
that,
when
doing
any
sort
of
deployment,
you
only
really
need
a
new
and
an
old
and
that's
exactly
what
this
mechanism
is
built
for.
If
you
use
just
the
default,
you
get
a
standard
Bluegreen
deployment,
as
most
of
you
are
familiar
with
with
the
kubernetes
deployment
resource.
B
If
you
do
a
canary
with
present
0,
you
get
what
we
call
a
pinned
mode
where
you
have
an
addressable
primary
and
addressable
default,
an
addressable
canary
end
point,
and
that
allows
you
to
send
traffic
or
experimental
traffic
to
the
new
endpoint.
But
customer
traffic
isn't
sent
there
yet
and
then,
as
you
and
the
third
option
is
canary,
which
is
this
percentage
here
in
between
the
two
specs
and
that
allows
users
to
or
that
allows
deployers
to
slowly
increment
the
traffic
to
transparently
pass
request
to
the
canary
deployment.
B
In
this
example,
we're
experimenting
with
an
iris
v2
instead
of
iris,
and
as
we
slowly
increment
this
canary
for
traffic
percent,
we
can
gain
confidence
that
our
deployment
is
not
going
to
break
our
users.
Eventually,
we'd
go
to
a
hundred
flip
the
canary
and
default
and
then
delete
the
canary
afterwards.
B
That
sounds
like
a
lot
of
complicated
functionality
and
it
is,
and
I
certainly
don't
want
to
implement
all
of
it.
One
of
the
most
amazing
pieces
about
kubernetes
is
that
all
of
these
is
that
kubernetes
itself
is
very
extensible,
but
so
are
the
layers
that
are
built
on
top
of
kubernetes.
Many
of
you
are
probably
familiar
with
sto
and
Kay
native,
which
is
which
are
projects
that
we've
chosen
to
be
our
foundational
layers
for
server
lists
and
networking
KF
serving
is
focused
just
on
the
machine
learning
problems.
B
B
Ml
surface
meshes:
why
do
we
need
a
service
mesh
for
ml?
Well,
like
we
said,
we
think
that
modern
production
functionality
is
necessary
for
all
work
loads,
including
ml.
A
service
mesh
is
typically
implemented
as
a
proxy.
That's
dropped
into
every
workload
that
proxy
controls
all
traffic
in
and
out
of
the
pot
and
allows
you
to
transparently
implement
a
whole
bunch
of
features
like
tracing
authentication,
HTTP,
upgrading,
Network
policy.
All
of
those
things
are
maybe
not
something
that
you'd
want
on
day
one,
but
on
day
100.
B
When
your
ops
team
comes
to
you
and
gives
you
these
requirements,
you
don't
have
to
do
anything.
You
can
keep
your
KF
serving
set
up,
working
and
just
start
applying
sto
policies
and
it
works
magically.
This
is
that
layering
you
saw
in
the
previous
slide
and
it's
really
critical
for
how
a
KF
serving
maintained
its
agility
and
provides
these
features
to
data
scientists
can
ativ
is
the
next
layer.
B
I,
don't
know
if
any
of
you
have
worked
with
K
native
before,
but
it's
a
fairly
complicated
resource
model
I
think
it's
the
right
resource
model,
but
it's
also
a
little
bit
too
much
for
data
scientists.
Most
people
only
need
their
top-level
resource,
which
is
a
service,
but
sometimes
you
might
need-
or
rather
the
the
underlying
resources
are
just
kind
of
overly
complicated,
so
we've
abstracted
K
native,
but
we
still
get
all
the
benefits
around
routing
and
queuing.
B
The
K
native
has
the
most
valuable
piece
of
K
native
for
us
is
the
KPA
or
the
request
based
autoscaler
if
you're
familiar
with
kubernetes
horizontal
autoscaler,
you're,
probably
familiar
that
you
can
set
a
CPU
or
memory
target,
and
once
you
exceed
that
target,
you
start
to
scale
up.
This
doesn't
work
that
well
for
a
lot
of
mo
workloads,
especially
ones
using
accelerators
if
you're
not
CPU,
bound
and
not
memory
bound.
How
do
you
know
to
scale
up?
B
You
could
do
something
like
Nvidia
does,
which
is
you
prove
Nvidia
provides
duty
cycle
which
is
a
GPU
metric,
but
that's
effectively
power
on
the
GPU,
and
it's
often
the
case
that
the
ML
model
isn't
using
the
full
power
of
the
GPU,
and
so
it's
really
hard
to
figure
out
exactly
what
metric
to
set
there.
The
request-based
autoscaler
for,
however,
works
really
well
there's
a
concept
of
desired
concurrency
and
actual
concurrency,
and
so
from
machine
learning
models
desired.
B
B
B
That
was
cane
that
was
KF
serving
vo
dot.
One
we're
now
on
KF
serving
ODOT
too,
and
we've
continued
to
listen
to
data
scientists
and
continued
to
look
at
the
industry
to
see
what
we
could
simplify
and
what
we
could
automate.
We
came
up
with
these
two
extra
concepts
of
transformer
and
explainer.
You
may
be
familiar
with
the
field
of
explanation,
which
is
where
you
send
multiple
requests
to
a
predictor,
and
you
perturb
the
input
several
times
to
see
what
predictions
come
back.
B
When
you
look
at
all
the
results
based
off
of
the
perturbed
input,
you
can
make
some
reasoning
or
you
can
reason
about
those
results,
to
figure
out
what
parts
of
the
input
were
most
important
in
getting
this
prediction.
It's
a
really
cool
field
and
there's
another
talk
on
this.
By
Selden
later
they
have
a
explainer
library
called
alibi,
but
we're
basically
just
an
architectural
component
of
an
explainer.
B
So
we
have
an
out-of-the-box
alibi,
explainer
implementation
that
can
be
configured
in
Y
amyl
or
you
can
provide
your
own
container
implementation
of
an
explainer
and
we
wire
up
the
routing
so
that
you
just
drop
it
in
in
the
spec
and
it
wires
up
automatically.
We've
also
seen
the
concept
of
a
transformer
which
is
incredibly
important.
Maybe
you
want
to
take
a
request,
clean
it
or
pre
process
it
or
translate
a
user
ID
into
a
bunch
of
features
about
the
user.
You
need
some
hooks
somewhere
and
often
machine
learning
frameworks.
B
Don't
give
you
a
place
to
execute
that
code?
We've
implemented
this
hook
in
KF,
serving
as
what
we
call
a
transformer.
So
the
request
comes
in.
We
translate
a
user
ID
into
a
whole
bunch
of
features
about
that
user.
The
explainer
then
takes
all
those
features.
Perturbs
them
sends
many
prediction
requests
the
pretty
hello,
the.
B
B
A
Next
I'm
going
to
talk
about
how
care
service
you
sow
our
Bloomberg
data
sense
platform,
so
Bloomberg
data
science
platform
is
for
Bloomberg's
machine
learning,
practioners
and
if
it
runs
on
a
computed
class
with
a
GPU
notes.
Bloomberg
has
a
sizable
number
of
data
science
and
AI
teams.
So
almost
a
lot
of
the
team
that
need
to
runs
your
infant
service
production.
A
A
Studying
models
on
production
is
not
a
unique
problem
to
Bloomberg,
as
Alice
just
mentioned.
Come
company
virus
different
company,
the
currently
doing
a
different
way
to
serve
models,
and
even
within
organizations,
teams
are
doing
basically
various
different
biome
itself,
and
so
we
reach
out
to
community
early
this
year
and
we
talked
about
with
Ellis
and
a
few
leaders,
and
we
started
K
of
serving
to
really
address
address
all
these
issues
by
providing
a
standardized
and
outbox
model
serving
across
different
model
frameworks
and
to
complete
a
have
a
complete
story.
A
We
also
support
a
model
experiment,
a
and
inference
graph
with
a/b
testing
and
multi-armed
bandit
and
Cave
serving
can
be
easily
drawn.
Our
current
existing
key
need
evidence
to
stack.
Oh
that's
a
pretty
in
nature
move
for
us
to
so
this
is
a
diagram
for
typical
lifecycle
for
Bloomberg
data
scientists.
So
there
are
scientists
spend
many
cycles
to
trigger
mode
of
long
as
your
local
terminal
environment,
to
dev
and
to
proud
to
achieve
proper
and
good
model,
and
and
many
of
the
data
science
actress
thinks.
Oh
I
have
a
model.
A
Now
my
job
is
done,
and
this
is
operational.
People's
problem
to
please
the
problem
about
to
production,
actually,
the
last
step
is
very
critical,
because,
in
order
to
realize
that
all
these
the
business
values
of
the
models,
we
have
to
observe
the
model
production
to
serve
the
real
production
traffic.
However,
this
is
a
very
challenging
task
to
a
lot
of
data
scientists.
A
So
we
found
out
a
lot
of
time,
like
a
lot
of
models,
are
not
getting
shipped
to
production
for
various
reasons.
For
some
they
they
don't
know
how
to
handle
the
production
load,
wasted,
recover,
which
required
latency
and
its
throughput,
and
for
some
some
settings
and
months
is
to
production,
lies
model,
and
sometimes
it's
just
a
figure
out.
They
are
not
very
confident
about
their
new
model,
and
so,
as
our
infrastructure
its.
We
really
want
to
take
this
responsibility
to
help
speed
up
the
production
lines.
A
A
So,
in
addition
to
all
the
the
problem
in
the
complexity
at
is
just
mentioned,
we
also
for
Bloomberg.
We
also
have
list
of
long
list
of
for
production
requirements.
We
want
to
support
a
production
user
with
24
hours
in
a
70s
report
support
and
we
want
a
safe
robot
in
robot,
low-back,
Pro
Street
procedure,
and
we
want
to
have
established
a
beaut
process
for
the
image
get
deployed
to
production
and
to
have
a
probe
approval
process
for
all
the
inference
service
department
once
inferences
it
brought
on
production.
A
We
also
need
to
have
other
logins
tracing
available
for
people
to
debug
their
problems
too,
as
well
to
monitor
the
latency
and
a
throughput.
We
also
need,
like
a
reliable,
alerting
we
invert.
The
inference
service
goes
wrong,
cave
serving
comes
into
picture,
and
it
actually
helps
us
to
adjust
a
lot
of
the
problems
on
the
list
since
caves
ovens
building
on
the
server
laser
framework.
A
It
abstractor
way
adopted
to
the
de
product
complexity
for
us
and
it
offers
canary
a
lot
strategy
and
it
also
provides
a
very
clean
and
Itakura
to
interface,
which
is
very
easy
for
people
to
review
and
which
answers
question
what
is
actually
deploy
to
production?
Are
you
rolling
out
a
new
model
or
you
change
some
configurations
or
changes
normally
instances
and
case
serving
also
expose
a
set
of
standard
metrics
for
latency
reports
and
concurrency,
so,
which
is
very,
will
be
a
critical
for
us
to
based
on
those
metrics
to
set
up
our
alerting.
A
So
the
most
important
use
case
is
to
we
want
to
standardize
automotive
serving
across
the
model
frameworks
and
also
we
want.
We
don't
want
each
team
to
revenge
the
area
where
to
build
their
own
model
server
over
the
time.
That
is
not
performance
and
hazard,
and
has
it's
very
not
like
a
transparent
to
what
is
actually
going
on
in
those
servers?
A
A
We
also
wanted
to
do
inference
on
news
Kafka
streamys,
where
all
the
news
comes
to
Kafka
and
we
wanted
to
really
return
in,
for
instance,
a
new
stories.
We
also
request,
like
a
GPU
impressed
with
tensorflow
and
paramount
to
meet
our
latency
and
a
support
requirement.
We
are
also
very
working
with
closely
with
community
to
build
out
GPU
sharing
solution
for
us
to
improve
the
utilization
of
GPUs
and
we
were
there's
a
need
for
a
be
testing
as
well.
A
So
well
goes
through
a
few
some
pros
spec.
We
use
cases
we
are
currently
using.
So
this
example
where
we
want
to
use
a
cave
settings
transformer
concept-
let's
just
talk
about
so
kept
serving
offers
a
SDK
where
you
can
easily
allows
to
implement
your
own
pre
and
post
processing.
So
you
extend
the
cave,
servings
base
class
and
you
can
hook
up
you
as
your
own
pre
and
post
processing
handler
once
you
you
build
in
Mitch
and
you
can
add
the
transformer
to
your
inference
service
pack.
A
Once
you
submit
to
the
inference
service,
you
wiii
cave
serving
x-rays
to
microservice,
my
name's
transformer
and
one
as
a
predictor
we
may
request
comes
in
it
will
hit.
The
transformer
first
invokes
your
pre
process
handler
transformer,
which
we
are
all
then
call
out
to
predict
her
to
get
the
prediction
result
back
and
again,
Nadeau
invokes
the
post
process,
which
often
the
time
people
use
it
to
deliver
a
more
user-friendly
response
back
to
the
end-user.
A
We
separated
a
concept
of
transformer
and
predictor
for
a
good
reason,
and
in
this
way
you
can
actually
keep
your
chance
mark
code.
While
you
want,
you
can
like
a
switch,
a
prettier
ways
more
performing
the
motor
server,
for
example
like
Onyx.
If
you
want,
and
also
you
can
like
a
scarer
transformer
and
particular
differently,
you
can
even
run
transformer
Brown,
CPU
and
particular
Allen
GPU.
A
So
model
explanation
is,
are
becoming
more
and
more
important
nowadays
on
the
production.
If
it's
all
races
are
so
alibi
is
a
open
source
with
from
certain
for
model,
explanation
and
interpretation.
It
is
supposed
support
a
multiple
use
cases
such
as
tabular
text
and
image
data.
It's
a
black
box
model,
acceleration
algorithm,
which
is
a
increment
based
on
the
paper
called
anchor
Maude
model,
agnostic
explanations.
A
This
is
the
tech
alibi
text,
extra
inspiration
to
us,
because
we
have
a
lot
of
NLP
use
case
where
we
we
can
easily
integrate
with
our
in-house
and
a
peer
model.
Server
sings
alibi,
Texas
already
needs,
is
to
have
accessibility
to
a
prediction.
Url
now
and
you
are
at
explain
around
to
the
inference
service
pack,
you
can,
you
can
actually
get
back
the
explanation
results.
So
when
you
call
the
explanation
and
part,
you
will
get
the
explanation
back
English
into
the
prediction
result.
A
A
We
also
have
a
used
case
where
a
lot
of
news
come
into
Kafka
and
we
want
to
have
a
inference
pipeline.
Where
goes
through
a
set
of
models
and
doing
the
name,
detection
and
duplication,
and
then,
after
the
last
step,
it
will
the
result
get
shuffled
on
to
out
Kafka
topic
and
then
which
can
be
further
consumed
to
for
the
downstream
end.
Clients.
A
On
country
we
are
also
making
appetizing
proposal
to
the
community,
where
we
were
in
conscious
a
canary
road
where
it
helps
attest
to
the
serve
your
service
performance
or
hers.
But
sometimes
you
may
also
want
to
test
the
model
accuracy
with
the
real
production
traffic.
So
here
we
want
to
want
to
introduce
this
concept
called
inference.
Router,
where
you
can
do
it
determines
the
routing
with
control
and
experiment
group.
So
the
group,
the
request
of
Ron,
contribute
a
lot
to
a
model
1
and
the
request
from
the
experiment
group
our
route
to
model.
A
So
after
we
are
able
to
choose
a
part
of
these
features.
Finally,
we
want
to
have
a
simple
and
reliable
the
process
for
data
scientists
to
deploy
your
inference
service
to
production.
So
we
wanted
a
size,
2
poster
the
machine,
only
source
code
and
as
in
14,
we
provide
a
Orchestra
CI
CD
pipeline
to
help
them
to
build
images
and
versioning
and
produce
the
motors.
So
the
design
state
can
then
create
a
cave
serving
the
inference
service
department
will
get
a
repo
to
write
a
cave
serving
spec.
A
We
just
showed
so,
as
you
can,
every
time
the
turret
you
can
update
the
image
version
or,
if
they're,
using
some
of
the
custom
cave
serving
or
pre-process
images,
and
whenever
they
wrote
new
models,
it
can
also
update
the
model
version
on
the
spec.
So
and
people
can
review
and
then
after
review
it
again
with
it.
A
A
So
the
demo
here
is
famous
like
a
mini
example.
So
what
I
want
to
do
here
is
I,
have
a
bucket
here
and
I
want
to
upload
a
digit
and
then
once
I
get
a
response
back
see.
I
have
zero
to
nine
here.
So
once
I
get
there
is
a
prediction
result
back.
It
will
automatically
put
into
this
one
with
the
packet.
A
So
in
order
to
do
this,
I
basically
implement
a
transformer
where
there's
a
pretty
process
and
post
process,
so
the
pre
process.
Basically
so
once
I
upload
to
the
image,
it
will
stand
a
Kafka
event
to
the
to
the
mean
your
publisher,
Kafka
event,
and
it
will
get
delivered
to
the
inference
arias
and
every
service
work,
how
this
process
function,
to
download
an
image
and
then
to
an
image
transform
after
you
get
to
the
result
back.
A
So
I
just
so
our
charge
with
the
product,
if
so,
I
keep
serving
from
the
notebook.
So
we
have
a
SDK
where
you
can
specify
the
transformer.
So
you
I
build
the
image
with
that.
Pre
and
post
processing
and
I
put
the
image
here
in
space,
because
my
crust
are
stingy
so
I
over
at
the
request,
our
image
and
then
I'm
going
to
create
a
specify
predictor
with
tensorflow.
So
I
have
a
model
chain
on
s3
bucket.
B
A
A
So
next
thing,
I'm
going
to
do
is
I'm
going
to
hook
up
to
the
Kafka.
So
as
I
upload
image,
it
will
publish
event
to
the
car
car
and
then
edit
basically
says
from
this
topic.
I
connect
to
this
cava
broker,
where
I
deployed
in
cluster
and
our
delivered
that
event
to
this
impress
service
which
they've
diminished.
A
A
A
B
A
A
B
B
Yeah
I
should
have
mentioned
that
yeah.
So
the
question
is
once
you
have
an
end
point
made:
what
does
the
request
look
like,
and
this
is
a
really
big
question
for
us?
Is
we
call
it
the
data
plane?
What
how
do
you
structure
a
request
for
prediction
and
it's
a
fairly
large
question.
Every
single
model
server
that
exists
in
this
history
right
now
is
similar
and
a
little
bit
different.
B
That's
working
on
this
problem
of
how
do
you
in
a
standard
way
send
a
request
to
any
framework,
and
that
would
enable
you
to
say
swap
out
a
psychic
learn
model
that
you've
started
implementing,
and
maybe
you
ran
into
some
performance
problems
and
did
better
with
PI
torch,
and
you
could
swap
that
out
under
live
traffic.
As
long
as
the
request
format
was
the
same
so
for
HTTP,
we've
decided
among
the
community,
including
open,
vino
and
tensorflow,
and
tensor
RT
at
Nvidia
and
a
bunch
of
other
I.
B
Guess
we
have
our
own
implementations
of
some
of
the
Python
ones.
We've
been
able
to
standardize
that
around
tensorflow
as
HTTP
protocol
and
if
you
go
in
our
Doc's,
there's
a
simple
structure
of
what
that
looks
like
and
it's
fairly
straightforward.
There's
some
pretty
serious
problems,
especially
with
encoding
images
into
JSON.
You
can't
put
bytes
into
JSON
and
images
are
typically
representative
bytes,
so
you
have
to
base
64
encoded
and
that
base64
encode,
decode
and
JSON
parsing
is
expensive,
and
this
is
where,
in
videos
really
helping
us
we're
talking
about
a
more
performant
interface.
B
So
we
would
expect
in
the
future
every
single
model.
Server
every
single
framework
would
support
two
interfaces,
one
very
simple,
JSON
one,
which
is
the
tensor
flows
existing
protocol,
but
you
could
just
curl
like
an
array
of
floats
if
you
wanted
to,
but
if
you
needed
to
do
something
more
performant
like
image
processing,
you
could
do
something
like
proto
before
flatbuffers,
that's
still
in
development,
so
that's
kind
of
the
current
state
of
things
we're
hoping
within
several
months
that
we
have
a
really
solid
story
around
what
the
performance
standardized
data
plane
is.