►
Description
Want to view more sessions and keep the conversations going? Join us for KubeCon + CloudNativeCon North America in Seattle, December 11 - 13, 2018 (http://bit.ly/KCCNCNA18) or in Shanghai, November 14-15 (http://bit.ly/kccncchina18).
Serving ML Models at Scale with Seldon and Kubeflow - Clive Cox, Seldon.io (Intermediate Skill Level)
Open source machine learning tools to manipulate and store “big” data and apply complex algorithms to this data have matured greatly in recent years. However, the final step in any successful machine learning project is to put the models into production with exposed APIs as well as monitor, scale and continuously update them. Kubernetes offers the ideal platform to build an ML deployment platform. This talk will review some of the challenges in deploying machine learning models and discuss Kubernetes focused open source projects (Seldon Core, KubeFlow) that provide solutions.
About Clive
Clive is the CTO at Seldon. He has a research background in Computational Linguistics and studied Speech & Language Processing at Cambridge University. Clive has spent the last few years developing Seldon’s algorithms and predictive platform and working on various machine learning projects in industry.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
Join us for KubeCon + CloudNativeCon in San Diego November 18 - 21. Learn more at https://bit.ly/2XTN3ho. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
B
And
just
a
shout
out,
we
also
run
tensorflow
in
London,
beat
up
so
any
of
you
guys
here
in
London
and
want
to
come
to
learn
more
about
tensorflow.
We
have
that
each
month
have
a
couple
of
talks,
both
quite
advanced
in
quite
beginner,
so
come
along
as
company.
We
do
two
things.
We
have
a
product
which
is
based
on
machine
learning,
deployment
on
kubernetes,
which
I'm
only
going
to
be
talking
about
today,
and
we
also
do
a
mild
consultancy,
especially
in
the
FinTech
area,
would
work
with
banks
on
doing
predictions
in
various
areas.
B
So,
for
today's
talk,
what
I'm
going
to
talk
about
is
we
should
landing
on
kubernetes?
Why
that's
good,
hopefully,
as
you've
seen
many
queue
flow
talks.
You
should
probably
believe
that
by
now
then,
focus
on
machine
learning
deployments
are
just
the
serving
an
inference
layer
and
talk
about
what
we're
doing
at
sell
and
call
and
talk
about
how
we
integrate
with
cue
flow
and
then
I
have
a
example
of
an
end-to-end
machine
learning
example,
which,
if
there's
enough
time
I'll
give
a
demo
of
we'll
see
what
we
get
soon.
B
So
one
thing
we're
trying
to
Excel
then
is
really
try
to
get
these
three
types
of
people
to
work
together
well
on
a
machine
learning
project.
So
you
got
the
data
scientist
obvious,
use,
doing
the
core
algorithms
and
data
analysis,
but
then
to
get
it
into
production.
They
gotta
work
with
like
a
DevOps
engineer,
and
these
people
sometimes
don't
have
a
good
understanding
of
machine
learning
and
don't
really
know
how
that's
going
to
work
and
also
the
data
scientist
doesn't
want
to
really
get
too
involved
with
maybe
the
DevOps
side.
B
So
how
do
you
get
those
two
sides
to
work
together
and
once
you
have
the
business
manager
who's
defining
the
project?
It
gets
the
RI
and
wants
to
make
sure
that
it's
going
to
work
well
and
these
three
people
have
to
work
together
well
for
a
machine
learning
project
to
actually
be
successful
and
really
that's
the
sort
of
goal
of
our
companies
to
try
and
make
these
people
work
together
with
various
certain
systems.
B
So
machine
learning
on
kubernetes
and
probably
don't
have
to
explain
too
much
about
that
or
why
it's
a
good
thing,
but
obviously
you've
got
the
GPU
a
sauce
aspect,
so
you
can
easily
push
your
jobs
to
the
proprietary.
That's
that's
needed.
Obviously,
the
fact
that
you
can
move
between
cloud
and
premise
is
grateful,
especially
our
customers,
in
banks
who
who's,
who
might
do
some
proof
of
concepts
on
cloud,
but
they
probably
want
to
do
on-premise
machine
learning,
adventure
for
various
reasons.
B
Then
you
get
all
the
stuff
up
but
offer
up
there
in
the
top
left,
which
you
guys
probably
know
about,
which
is
good
for
any
software
product
was
a
project,
but
it's
also
good
for
machine
land.
It's
all
the
are
back.
You
know
making
sure
that
people
working
on
machine
lighting
projects
just
have
the
quotas
and
resources
they
need.
They
could
scale
up
what
they're
doing
health
checks,
what
they're
doing
and
all
be
separated
out
on
a
communities.
Cluster
says
many
cute.
B
There's
many
projects
out
there
that
already
machine
many
bodies
running
on
top
of
kubernetes
park
and
Jupiter
amongst
many
others,
and
we're
part
of
that
ecosystem.
That's
growing
with
machine
learning
on
top
of
kubernetes
as
coop
flow
as
well,
so
now
I'm
going
to
talk
about
machine
learning,
deployment
and
southern
court
in
particular.
So
these
are
some
of
the
challenges
or
goals
that
were
trying
to
solve.
First,
the
core
deployment
aspect.
B
So
a
lot
of
this
has
really
helped
by
walking
on
top
of
kubernetes
a
lot
of
these
things
here
actually
solved
by
curb
measures
itself
in
terms
of
launching
a
scaling
up,
but
also
as
many
ways
you
want
to
update
your
running
machine
learning
project
in
terms
of
various
ways
you
might
want
to
update
it
in
real
time.
Not
just
doing
rolling,
updates
and
I'll
discuss
some
of
those
and
that's
part
of
what
we're
trying
to
do
in
our
project
as
well
and
obviously
healthchoice
in
recovery.
B
We
want
to
do
everything
as
a
ssin,
so
obviously
infrastructure,
optimizations
that
you're
saving
money
on
machine
lane,
because
me
showing
machine
learning,
prediction
covers
you.
The
infant
site
could
be
very
resource
intensive,
and
so
you,
if
you
want
to
make
a
profit
on
on
your
machine,
early
applications
out
there,
you
need
to
get
that
right.
I'm
obviously
latency,
especially
in
the
FinTech
world
and
various
areas
where,
as
low
latency
is
required,
that's
needs
to
be
tried
and
optimized
as
much
as
possible
and
also
the
other
side
free
put
some
projects.
B
You
want
to
get
maximum
throughput
in
terms
of
the
predictions
going
for
your
system
and
so
that's
key
and
then
every
form
of
actual
model
optimization
as
well.
So
if
you
might
have
multiple
models
trying
to
work
out
which
model
is
the
best
trying
to
change
them
in
real
time
in
terms
of
pushing
out
traffic
to
the
correct
model
and
I
go
through
some
examples
there
and
then
finally,
obviously
you
deploy
your
machine
learning
inference
stack,
but
you
need
to
then
tie
it
into
the
business
who's
going
to
be
using
those
predictions
to
do
things.
B
So
it's
part
of
seldom.
We
automatically
expose
rest
and
gr
PC
in
terms
of
what
we're
doing
and
in
future,
we're
going
to
look
at
more.
I'm
a
synchronous
methods,
which
is
a
some
of
our
customers
need,
was
that
still
early
stages
and
we're
not
sure
how
that's
going
to
go,
and
it's
probably
interesting
interactions
with
cue
flow
there.
In
terms
of
how
that
will
work,
and
then
the
whole
sort
of
management
side
is
key
as
well,
so
all
the
aspects
of
all
the
thing
and
versioning.
B
So
how
do
you
know
that
what's
changed
and
from
the
previous
model
update?
Maybe
that
needs
approval
before
you
actually
put
it
out
into
production.
What
versions
of
the
data
provenance
has
great
talks
about
packing
them
tomorrow
yesterday
and
I
think
they
fit
in
key
for
that
area,
and
we
definitely
want
to
use
that
and
it's
great
well
with
them
all
forms
of
monitoring.
B
Is
my
machine
learning
working
well
in
terms
of
the
accuracy
etc
and
see
ICD,
and
things
are
tired
to
get
hops,
and
I've
explained
about
how
that
is
viewed
in
terms
of
machine
learning
deployment
to
do
those
aspects,
then
two
things
I'm
going
to
discuss
him
slightly
more
detail.
One
thing
we
definitely
wanna
do
is
allow
data
scientists
to
use
any
of
the
tools
that
they're
using
now
and
not
be
restricted.
So
we
just
want
to
focus
on
deployment
the
serving
in
front
side,
and
they
can
use
python
to
do
tends.
Flow
sky
can
learn.
B
They
might
want
to
do
their
models
in
our
spark
h2o.
It's
anything
that
using
that
all
commercial
toolkit,
so
just
the
training
that
can
continue
as
they
were
doing
now
and
we
want
to
handle
the
serving
side
and
then
we
want
to
handle
the
actual
service
mesh
of
the
one-time
prediction
aspect,
so
not
just
the
model
but
also
routing
requests
so
doing
a/b
tests,
multi-armed
bandit-
if
you
don't
know
that
I'll
explain
about
that
later
and
give
an
example
transformations
so
feature
normalization.
B
You
might
want
to
put
that
as
a
separate
docker
image
and
dokdo
Kermode
you'll
to
do
that.
Maybe
also
you
want
to
wash
your
ensemble
results
from
multiple
models
to
come
back
together
to
get
a
single
result
and
metrics.
So
the
things
that
you
want
to
do
things
like
concept
drift,
so
is
my
model
accuracy
decreasing
in
time.
So
maybe
it's
because
there's
different
data
coming
in
in
real
time,
which
was
not
the
same
as
they
try
trained
it
on.
B
So
maybe
I
need
to
retrain
my
model
and
guess
again
get
alerts
there
or
stuff
like
out
lie
detection.
You
know
other
strange
data
requests
coming
in.
Can
I
kind
of
have
these
flack
your
mobile
to
add
modules
into
your
one-time
wofully
that
should
provide
these
pieces
of
functionality
and
every
aspect
of
security
obviously
is
key.
B
So
this
is
a
slide
from
cue
flow.
Just
this
really
make
it
clear
where
we're
situated
so
I
see
the
data
science
steps
as
you've
seen
puppy
seed.
There's
many
steps
in
this
flow
that
you
need
to
go
through
in
a
data
science
project,
and
probably
all
of
these
have
a
feedback
loops
to
previous
steps,
as
you
iterate
around
in
terms
of
what
we're
doing
we're
really
just
doing
the
rollout
serving
monitoring
of
a
machine,
lender,
deployments
and
cue
flow
is
obviously
a
whole
ecosystem
providing
a
set
of
easy
packages.
B
B
So
what
you
need
to
do
to
get
running
with
Selden
free
steps.
So
we
have
helm
charts
to
deploy
once
you've
got
your
kubernetes
cluster.
We
also
have
K
sonnet
registries.
We
are
well
in
case
you
aren't
reading.
Also
one
is
part.
We
also
have
a
package
as
part
of
queue
flow,
so
you
can
easily
install
it
as
part
of
the
queue
flow
registry.
Secondly,
you
will
have
your
own
models.
B
Your
own
one
x,
and
you
need
to
then
package
shows
up
as
images,
so
we
can
handle
that
you're,
not
so
micro
service
graph
to
actually
manage
those
at
one
time.
So
we
provide
different
methods
I'm
going
over
one
method.
We
provide
to
make
that
easy
for
developers
so
that
that's
using
openshift
source
2
image
to
take
a
source
and
turn
it
into
an
image,
and
then
finally,
we
have
a
custom
resource
that
defines
your
graph
that
you
want
to
have
at
one
time
and
then
obvious.
B
You
can
just
use
cube
control
to
the
actually
deploy
out
to
the
graph
and
so
deploy
that
to
kubernetes.
So
this
is
the
architecture
of
Selden
Corps,
so
you
can
discuss
on
the
right.
You,
you
push
your
custom
resource
into
the
cluster.
We
have
operator.
That's
running
in
the
cluster
the
spots
these
things
coming
in
and
then
crates
the
underlying
kubernetes
resources,
deployment,
graphs,
deployment,
services,
pods,
etc
to
actually
manage
that
graph
and
handle
changes
to
it.
B
So
you
can
do
oh
or
for
whatever
authentication
you
need
to
plug
in
there.
So
it's
pretty
simple
things:
pretty
easy,
how
it
works
and
to
set
up
and
get
going.
So,
as
I
said,
we
want
to
have
reasonably
complex
one-time
graphs,
so
not
just
model.
So
obviously,
models
is
the
thing
you
all
expect
in
terms
of
machine
learning
at
one
time
secret.
Your
tensor
flow
model
etc,
and
we
want
to
allow
that
we
will
also
want
to
allow
Reuters,
which
we
call
Z's.
B
These
are
the
false
or
abstract
types
that
we
allow,
so
Reuters
basically
take
a
request
and
decide
where
it's
going
to
go
further
down
the
graph,
so
that
would
cover
things
like
a
be
testing
or
multi-armed
bandit
swear,
you're,
looking
at
which
model
is,
is
working
best
and
pushing
resources
to
the
best
performing
model.
We
also
have
combiners
which
take
the
responses
coming
back
up
from
the
models
and
join
them
together.
B
So
you
can
use
that
to
obviously
ensemble
different
models
into
different
responses
from
models
into
a
single
result
and
transformers
which
is
sort
of
generic
component,
where
you
might
with
to
use
that
to
feature
that
feature
normalization.
So
it's
taking
the
request
and
changing
it
into
something
else.
It
will
also
contain
the
response
and
change
it
to
something
else,
as
goes
as
it
goes
back
out,
and
that's
where
you
would
handle
things
like
concept
drift
or
out
lie
detection.
B
So
these
modules
might
add
a
little
extra
data,
actually
some
metadata
to
your
response,
as
it
goes
back
up
through
the
graph,
so
to
clarify
that
I'm
just
going
to
show
you
some
examples
of
how
that
might
happen,
so
you
could
be.
Two
parts
is
like
client
components
which
is
the,
which
is
the
components
that
you
build
and
then
there's
third-party
components
which
might
be
a
stuff
you
take
from
Seldon,
core
or
third
parties
provide
that
so
the
simple
thing:
okay,
you've
got
a
model.
We
can
manage
that.
B
Obviously,
and
it's
connected
your
API
GRP
cos,
you
send
a
request,
gets
back
response,
but
maybe
your
data
scientist
creates
a
new
model
that
they
want
to
test
out.
So
it
won't
have
an
a/b
test.
You
want
to
do
a
rolling
update
of
that
model
and
to
do
a/b
test
to
test
out
the
accuracy
and
success
of
each
each
model.
But
then
maybe
you
want
to
do
something
more
complex.
B
Maybe
we'll
have
four
models
and
you
want
to
run
a
multi-armed
bandit,
and
so
you
have
to
take
the
requests
as
they
come
in
and
decide
which
of
those
models
is
performing
best
and
push
push
all
your
requests
to
that
model
while
at
the
same
time,
maybe
monitoring
every
model
so
doing
more
complex
things
in
that
area.
Maybe
you
want
to
separate
out
the
feature
normalization
just
because
all
those
models
are
using
the
same
set
of
features.
You
want
to
separate
that
into
out
into
a
separate
image
and
handle
that
as
part
of
a
graph.
B
But
it's
hard
to
explain
those
to
stakeholders
such
human
beings
or
why
that
prediction
was
made
for
a
particular
input
data,
so
you
can
use
various
techniques
like
this
there's
one
called
lime
and
various
other
techniques
out
there
that
you
can
use
to
try
and
give
more
human,
readable,
understandable
results,
I'm
human,
understandable
explanations
as
to
why
your
models
running
so
you
might
want
to
tie
that
in
normal.
Actually,
these
things
are
quite
so
resource
intensive,
so
you
probably
wouldn't
put
that
in
production
and
probably
have
a
site
cluster
running
to
where
you
could
say.
B
Okay
for
this
particular
request,
why
did
it
give
that
response?
So
this
is
the
sort
of
thing
and
you
want
to
sort
of
iterate
between
these
inbuilt
time
to
update
them
and
with
no
downtime,
so
I
know
how
readable
that
is,
but
I'm
going
to
show
the
core
blocks
of
our
custom
resource
of
how
you
define
the
graph.
So
the
first
thing
is
you
give
the
graph
definition,
so
that
would
be
what
I
showed
you
on
the
previous
slide,
where
you
define
how
your
components
fit
together
and
your
graph?
It's
very
simple.
B
It's
just
a
single
model,
but
obviously
more
complex
in
terms
of
the
other
things
I
showed
there.
Then
you
have
a
pod
temple
expert,
which
is
really
just
simply
a
kubernetes
report.
Template
expects
you
can
put
everything
in
there
that
you
have
in
a
normal,
combat
communities,
port
template,
spec,
so
the
image
and
all
the
stuff
you
might
want
to
connect
to
it
like
the
volumes,
so
resource
requests
like
GPUs,
etc.
B
So
that's
just
standard
kubernetes
port
templates,
but
then
you'd
obviously
set
the
number
of
replicas
how
many
copies
of
this
go
after
you
want
to
run.
And,
finally,
the
fact
that
you
can
also
allow
a
list
of
these,
which
we
call
predictors,
that
those
parts
down
below,
and
so
you
cannot
just
have
one
but
multiple
sets
of
those
running
in
terms
of
your
graph
and
I'll,
explain
why
in
the
next
slide
or
things,
this
is
where
it
ties
into
how
you
might
want
to
do
updates
to
keep
the
complete
resource
isolation
between
u1
and
graphs.
B
One
is
obviously
Canaries,
so
here
we
assume
you
have
a
model,
one
one
model
running
with
ten
replicas
and
we've
deployed
it
and
crated
these
components
underneath.
But
you
want
to
test
a
new
model
and
check
out
whether
it's
going
to
blow
up
or
it
was
going
to
be
okay,
so
you
just
have
one
replica
of
that
may
be
running
handling
production
traffic
until
you're
happy
and
then
you
do
it
a
rolling
update.
So
we
can
handle
that
and
that's
where
you'd
have
these
two
grass
grass
completely
isolated.
B
So
it
means
you
discontinue
a
connected
to
the
existing
production
graph
and
then
maybe,
when
you're
happy,
you
do
a
rolling
update,
Bluegreen
deployments.
That's
not
part
of
the
current
open
source,
but
we
plan
to
handle
that.
So
obviously
you,
you
have
a
say,
ten
replicas
running
of
a
single
model,
and
then
you
want
to
completely
set
up
a
separate
model
of
ten
replicas
once
that's
all
set
up,
because
sometimes
machine
learning,
a
predictors
take
a
long
time
to
load
all
the
parameters,
etc.
B
So
that's
one
part:
that's
the
graph
that
we
deploy
now.
The
other
part
is
how
do
you
make
it
easy
for
developers
to
take
their
one
time
and
images
and
one-time
source
code
for
the
machine?
Learning
code
and
put
it
into
an
image,
so
they
can
do
it.
These
are
the
two
steps
they
have
to
do.
One
is
there
to
dock,
or
is
it
to
put
it
into
a
container
and
second
they
have
to
expose
the
API.
So
it
follows
our
less
than
gr
PCM
API
s--,
so
they
can
do
that
themselves.
B
Basically,
so
you
have
your
source
code
that
has
some
source
code
on
the
right
for
a
and
it's
not
always
always
classifiers
you've
got
a
predicted
method.
It's
loading
its
model
parameters
from
someplace
on
disk,
and
you
want
to
turn
that
into
image.
Basically,
you
choose
one
of
our
builder
images.
This
is
a
source
to
image
term,
so
we've
got
to
Python.
B
So
you
use
the
magical
source
to
image,
to
actually
create
it
and
I'll
give
a
quick
example
here.
So
this
is
for
wrapping
pie
for
models.
You
can
do
that
on
one
line,
so
he
gos
I,
build
on
the
current
directory
you're
saying
I
want
to
use
the
Selden
Python
to
build
it.
Image
and
I
want
to
create
an
image
called
my
repo
iris,
PI
classifier,
so
you
provide
your
source
code.
This
is
your
prediction
source
code.
B
Remember
not
the
training,
just
the
prediction
source
code,
you
put,
you-
can
provide
a
set
of
requirements
which
would
then
get
some
pipe
installed
in
this
case,
ask
I
can't
learn
inside
PI
and
then
you
just
give
us
some
information
so
that
we
understand
how
to
join
these
bits
together.
So
this
key
can
even
be
provided
on
the
command
line
or
next
to
your
source
code
and
odorous
to
our
environment,
location,
see
you
say
what
is
your
model
names
that
ties
it
into
the
source
code
number
on
the
left?
B
And
you
say
what
API
types
you
want
rest
or
G
IPC,
and
what
is
this?
This
is
a
model
because
obviously
we
have
everything,
as
I
said
like
Reuters
and
ensembl
errs
so
Reuters
and
combiners
and
transformers
as
well
and
I'll
go
for
this
very
quickly,
but
we
also
have
our.
So
you
can
do
the
same
thing
with
our.
So
you
have
our
code.
B
You
can
give
a
set
of
dependencies
and
you
can
say
how
to
join
together
and
you
use
our
our
build
a
package
and
we
do
it
the
same
for
Java
javis,
bit
more
of
prescriptive
at
the
moment
and
you
in
to
create
a
spring
boot
app,
and
then
you
use
our
library
southern
call
rapper
to
then
have
the
various
parts
that
you
can
then
implement
to
manage
to
add
your
code.
So
in
this
case
implements
the
Zelda
model
Handler
and
you
do
the
same
thing
and
then
it's
just
one
line.
B
So
this
point
is
this
can
be
then
put
as
part
of
your
CI
CD
pipeline
very
easily,
so
that
as
code
gets
changed
and
in
your
models
they
can
just
go
through
it
and
package
them
up
as
images
and
push
them
so
just
to
clarify
the
steps.
So
one
is
you
want
to
package
up
your
runtime
graph.
You
define
your
graph
in
your
custom
resource,
deploy
it
onto
kubernetes
and
then
go
through
that
loop
of
doing
updates,
etc.
B
As
you
go
along,
so
we
have
two
external
API
so
that
we
provide.
So
this
is
how
you
tie
into
your
business
apps,
so
one
is
obviously
predict.
So
this
takes
various
types
of
payload,
so
you
can
just
have
a
tensor
which
is
just
a
shape
set
of
floats
or
you
can
have
like
an
NDA
which
allows
you
to
handle
to
a
multi
type
data.
B
So
that
would
allow
you
then
to
send
in
your
request,
you
get
back
a
response.
So
in
this
case
like
this,
is
the
EM
NIST
handwritten
digit
classification,
sending
a
five
image,
and
it
said
it's
a
five.
We
then
also
have
a
feedback
API.
We
can
say
whether
the
model
got
it
wise
for
wrong.
So
you
can
send
them.
This
is
a
request.
This
is
the
response.
I
got
back
and
then
you
can
send
in
I'm
a
reward,
just
say
whether
it
got
it
wired
to
a
wall.
B
So
in
terms
of
our
roadmap,
so
we
will
definitely
focus
on
so
low
latency.
So
obviously
I've
shown
you
lots
of
complex
graphs
there,
but
definitely
there's
obviously
a
lot
of
use
cases.
We
just
have
a
single
model
and
that's
just
it
and
there's
a
lot
of
work,
especially
from
people
like
in
video,
unlike
ten
to
Artie
and
other
things,
they're
doing
to
really
optimize
use
of
GPUs,
and
we
want
to
make
that
easy
for
Dever.
So
data
scientists
to
use
stuff
like
predictive,
batching
I
would
definitely
wanna
have
a
look
at
that.
B
That's
in
tensorflow
serving
and
that's
great,
it's
great,
and
we
want
to
have
that
as
part
of
our
car
code
as
well,
and
so
everything
around
the
area
of
optimizing
single
model
things.
So
you
can
have
the
flexibility.
I've
also
data
will
encode
provenance,
so
won't
integrate
in
with
people
at
pachyderm.
So
we
can
get
that
aspect,
and
then
things
like
get
ops
to
allow
your
code
to
be
clearly
defined,
and
then
we
definitely
handle
sort
of
things
like
distributed
graphs.
So
not
just
at
the
moment.
B
B
Obviously,
they
also
use
C
IDs
for
tensorflow
and
pi2
option,
etc.
So
that
there's
a
great
so
compatibility
so
compatibility
there
in
terms
of
how
they
are
solving
of
things
and
the
terms
of
using
queue
flow.
Cs,
obvious
you've
seen
in
the
last
talk,
if
you
were
here
and
how
you
can
use
case
on
it,
to
include
the
packages
you
need.
So
here
you
include
tens
flow
serving,
but
you
just
can
switch
it
out
and
you
sell
them.
B
B
Your
one
time
parts
also
the
also
the
in
terms
of
training
parts,
quite
images
for
them
push
them
out
onto
a
cluster
with
cue
flow
on
to
do
the
training
just
in
tests,
obviously,
and
then,
as
new
images
are
created
that
could
be
pushed
into
your
resource
repos.
That
would
then
be
updated
with
new
image
versions
and
then
push
that
out
onto
a
cluster
for
production.
So
for
that
final
stage,
probably
things
like
we've
worked.
B
So,
yes,
a
quick
end
to
an
example
of
how
that
would
work
in
production,
so
this
is
using
the
classic
as
m-miss
classifier
and
so
you've
got
handwritten
digits
and
you
need
to
classify
them
into
zero
or
night,
and
you
can
find
this
example
on
cue
flow
example
Selden.
So
it
gives
it
end
to
end
example.
You
can
work,
for
it,
takes
about
20
minutes,
probably
to
set
up
and
run
all
the
whole
stages
someone's
on
GK
at
the
moment.
So
it's
quite
easy.
B
B
So
they
can
be
one
in
southern
core
and
finally,
once
you've
done
that
you
push
it
out
and
maybe
you
deploy
a
single
model.
Hopefully
I'll
show
one
model
to
probably
turn
it
into
an
a/b
test,
with
a
loading
update
and
then
do
a
multi
unbanned
it
and
then
go
through
that
ever.
This
also
uses
all
the
model
parameters
to
push
to
NFS
for
that.
So
let's
just
go
so
yes,
so
here's
the
repo!
It's
got
three
different
models.
B
I
won't
go
into
the
code
now,
but
there's
three
models:
a
tensor
flow
model,
skycat
learn
model
and
our
model
for
classifying
digits
and
there's
training,
which
I
won't
do
now.
Obviously
that
takes
some
time,
there's
a
training
notebook
that
goes
through
the
Argo
code
for
actually
training
each
of
those
models.
So
there's
a
auga
workflow
for
each
of
the
models
in
this
case
that
creates
the
image
form
from
this
source
code
turn
to
quite
an
image
and
then
pushes
it
out
and
transits
this
case.
It's
training
the
tensorflow
model,
using
it
other
TF
job.
B
So
that's
all
there
for
each
of
the
models
and
you
can
change
those
parameters
to
actually
use
your
own,
get
repo
and
push
your
own
docker
hub
repo.
So
in
terms
of
serving
so,
we've
already
got
a
model
running
so
as
part
of
the
southern
core,
we
have
a
standard,
I'm
gonna
find
a
dashboard,
so
here
you've
got
the
number
of
requests
per
second
running
through
them
up
this
one
model.
This
is
the
tensor
flow
model.
This
is
actually
the
accuracy
of
the
model.
Cuz
we're
actually
the
running
code
that
some
testing.
B
This
is
actually
saying
is
to
sending
out
random
digits
to
the
model
and
then
checking
when
it
comes
back,
whether
the
model
got
it
right.
So
it's
running
at
about
this
is
actually
not
that
great
for
emili´s,
but
I
wasn't
spending
much
time
in
this
example
for
to
get
high
high-performing
models,
they're
just
really
example,
models.
This
is
running
at
90
percent
accuracy
and
here
there's
natural
various
to
request
latency
to
go
through
so
in
terms
of
serving
so
what
we've
got
now.
B
So
here's
like
a
rest
request
for
two
got
that
one
right
or
you
could
send
in
GOP
and
that's
all
part
of
the
code.
You
can
see
how
easy
it
is
to
integrating
rest
and
gr
PC
into
your
code,
and
so
the
second
part
is
the
load
test
running
and
that's
already
running
there.
So
you
see
is
pushing
stuff
through
onto
the
models
and
now
what
we
can
do
is
we
can
turn
it
into
an
a/b
test.
So
I'll
just
apply
that
straightaway.
So
that
will
be
so.
B
It's
changing
the
actual
resource
definition
for
for
this
setup.
So
this
is
basically
what
we're
going
to
turn
it
into
we're,
going
to
turn
it
to
a
random
a/b
test
with
two
models.
We
have
the
tensor
flow
model
and
the
sky
Kettler
model
both
with
rest
endpoints.
So
what
does
that
look
like
in
terms
of
the
actual
CRD
customer
we
saw
so
now?
We've
changed
it.
So
we've
got
two
sections
here
in
the
containers.
Simply
the
other.
B
The
original
tensor
flow
model
and
we've
got
the
new
sky,
can't
learn
model
here
and
then
the
graph
definition
is
a
bit
different.
So
you've
got
the
random
a/b
test
with
a
few
configurations
of
how
much
traffic
you
want
partial
to
each
model
tends
flow
model
and
the
S
scalar
and
model
there.
So
let's
just
have
a
look
to
see
if
that's
started
up.
B
So
yes,
the
previous
one
was
terminating
now
and
it's
got
Linux
running
the
new
one,
so
we
can
send
in
some
requests
to
that
and
see
what
that
looks
like,
or
you
can
also
check
from
the
status
of
the
cell
deployment,
whether
it's
ready
or
not
and
I'll,
give
you
say,
c1
replicas
replicas
available.
So
now
we
can
send
an
another
worse
request,
and
here
all
you've
seen
that's
different.
Is
it's
given
as
a
extra
piece
of
metadata?
Which
of
the
which
are
the
two
routes
is
its
it
went.
B
This
is
one
probably
if
we
want
it
a
few
times,
we'd
get
the
other
one.
If
we're
lucky
50%,
there
were
three
times
as
a
Laker,
not
yet,
and
so
just
to
prove
that
we'll
just
run
a
test
to
run
a
hundred
requests
through
that,
and
we
can
also
just
update
the
graph
on
our
dashboard,
and
we
should
see
now
that
we've
got
two
models
going
through.
So
here
we
got
the
s
Keller
and
model.
B
This
just
started
and
we
got
the
new
version
of
the
the
tensor
flow
model
running
this,
so
that
the
update
you
see
this
s,
kilo
model,
actually
accuracy,
is
really
low.
Only
25%
here
and
you
see
the
traffic
is
now
split.
So
what
was
going
at
50
requests
per
second,
and
now
it's
not
till
25
on
each
so
yeah.
So
here's
an
example:
we
sent
100
requests
and
we
just
that
those
have
been
split
between
the
two.
B
These
are
the
two
end
points
1
or
0,
and
so
we've
got
our
traffic
split
and
that
was
done.
A
rolling
update
with
a
low
touch
running
with
no
downtime,
so
now
I'm
going
to
apply
more
complex
things.
A
multi-armed
bandit
so
I'll
apply
that
as
it
will
take
a
few
seconds
to
actually
start.
So
what?
If
we
look
at
straight
through
what
the
graph
is
now
we're
going
to
have
three
models?
So
we've
got
the
sky,
can't
learn
model.
B
It
tends
to
flow
model
on
the
arm
model,
they're
all
West
and
put
as
part
of
our
graph
and
we're
going
to
swap
out
the
random
a/b
test
and
put
in
a
multi-armed
bandit
here.
So
that's
going
to
look
at
the
feedback
coming
back
from
the
model
and
see
which
one
of
these
models
is
I
should
be
forming
best
and
then
push
traffic
to
the
one
that's
performing
best
and
just
keep
a
small
amount
of
traffic
on
the
other
ones.
B
So,
let's
just
see
where
that
is
so
now
yeah
we've
got
the
previous
one
is
terminating
and
we've
got
the
new
one
running
so
I'm
going
to
do
a
little
test,
which
will
take
a
second,
but
let's
just
run
a
test
through
so
now,
you've
seen
its
change
the
metadata
coming
back.
This
is
the
epsilon
greedy
Reuters
that
it's
taken
root,
two
for
this
request
and
so
I'm.
B
Just
going
to
send
in
a
hundred
requests
to
that,
and
we
can
also
just
check
the
graph
on
our
dashboard,
so
that
should
hopefully
be
changing
now
as
well.
So
now
we
should
have
three
models
running:
what's
got
it
so
we've
got
the
new
one,
which
is
the
R
model.
That's
coming
up.
We've
got
the
S
killer
model.
That's
there
previously,
and
we've
got
the
previous
deep,
deep,
deep,
M
this
model,
and
this
is
the
epsilon
we
did
that's
also
getting.
What
is
the
overall
reward?
It's
saying
all
those
are
there.
B
You
see
they're,
actually
quite
quite
different
in
terms
of
accuracies
and
in
terms
of
the
number
of
requests
they
serve.
The
our
model
is
actually
very,
very
slow
in
terms
of
what
it
can
serve.
So
here
we've
seen
so
here.
It
shows
that
the
most
young
baddest
gives
a
different
output.
So
basically
you
see
it's
quickly
chosen
that
the
tensorflow
model
is
the
best
and
it's
sending
all
the
traffic
to
the
tensorflow
model,
while
it's
still
sampling
from
the
other
two
models,
because
you
know
things
might
change.
B
In
some
circumstances
of
you
have
a
shopping
app.
Maybe
things
in
the
evening
or
better
for
one
model
on
the
wild
another
day,
so
you
can
make
this
all
dynamic
flow
between
the
models,
so
yeah
and
I.
Think
that's.
Basically,
if
we
go
back
to
the
presentation,
so
that's
showing
that
and
you
can
try
this
out
I'm
on
github
as
I
said
what
I've
shown
there,
and
that
is
it.
So.
Thank
you
very
much.
B
Certainly
we're
looking
for
the
next
stage,
because
what
the
moment
we're
so
synchronous,
so
sorry,
the
question
was,
will
be
support,
rabbit
all
0
mq.
So
at
the
moment
we're
looking
at
synchronous
requests
so
in
the
future.
We'll
want
to
look
at
asynchronous.
Definitely
we're
not
sure
what
direction
will
take
I
mean
what
obviously
want
to
be
as
flexible
as
possible.
B
The
honest
answer
is:
I,
don't
know
which
which,
which
is
it,
which
is
the
best
way
to
go
that
for
that
point,
and
it's
all
also
be
interested
to
see
how
queue
flow
is
handling
sort,
a
synchronous
requests
and
in
theirs
and
stuff
like
pachyderm
and
stuff,
and
how
that
would
fit
in
to
the
overall
pipelines
there.
So
yeah
I
don't
know.
B
So
the
question
was:
what's
what
is
the
recommend
is
still
wish
for
models?
I'm,
you
mean
the
models
will
also
the
model
parameters
or
both
yeah
I'm.
So
the
model
parameters
I
mean
I.
Suppose
you
could
use
whichever
is
most
appropriate
in
this
example.
It
was
like
NFS,
but
I
mean
maybe
use
it.
Other
storage
sounds
I
mean
suppose
you
could
use
cluster
FS.
If
you
want
so
distributed
storage,
you
know
anything
that
makes
most
sense
and
yeah
also
object.
Storage
in
terms
of
the
actual
models
yeah.
These
are
all
put
in.
B
To
get
so
I
mean
that's
sort
of
works
for
easily
in
terms
of
the
guitars,
in
terms
of
you
know,
tying
in
to
your
CI
CD,
so
I
think
yeah.
That's
what
we've
done
here.
It's
not
tied
into
CI
CD
here,
but,
as
you
can
see,
you
could
really
easily
extend
it
to
push
so
I
have
commit
hopes
that
could
easily
start
off
your
CI
CD
hi
yeah.
B
Sure
was
what
are
we
gonna
spoil
model
DB
I
haven't
looked
into
that
yet,
but
it
sounds
like
a
good
idea
sounds
like
the
summer
should
be
part
of
coop
flow.
Maybe
you
should
chat
with
Jeremy
over
that
so
yeah,
no
definitely
I'm
gonna,
be
great.
I
mean
obviously
we're
trying
to
get
like
an
end
to
end
set
of
components,
and
that
would
be
great
in
terms
of.