►
From YouTube: A Tale of Two Worlds: Canary-Testing for Both ML Models and Microservices - Jörg Schad
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
A Tale of Two Worlds: Canary-Testing for Both ML Models and Microservices - Jörg Schad, ArangoDB & Vincent Lesierse, Vamp.io
With the rapid and recent rise of data science, organizations are leveraging Cloud Native tools, especially Kubeflow for Data Science. One of the big challenges is how to deploy models in productions using similar practices like A/B testing and Canary-releasing which have proven successful for microservices. How to easily test and update your data models to production without impacting users? These are typical challenges a data-scientist will encounter when self-deploying and -managing the lifecycle of data models in production. In this talk Vincent Lesierse and Jörg Schad are going to show how experiences learned from releasing Microservices on Kubernetes can be applied to the world of ML Models, and where the deployment and lifecycle management of these ML Models differs from Microservices.
https://sched.co/MPau
A
Thank
you
so
much
for
the
introduction
and
welcome
to
our
talk
tale
of
two
worlds.
So
what
we're
going
to
be
talking
about
today
is
basically
here
at
cube,
con
I
guess,
most
of
us.
We
have
experience
with
some
kind
of
model
deployments
and
we're
actually
going
to
talk
about
the
similarities,
to
deploying
machine
learning
models
which
might
be
something
new
for
some
of
us
at
least
so
with
me
here
is
the
Vincent
yeah
I'm.
B
A
So
I'm
the
head
of
engineering
and
machine
learning
over
at
Arango,
DB
and
prior
to
that
I
actually
built
a
lot
of
machine
learning
infrastructure,
both
satsuki,
which
is
like
a
healthcare
I
based
start-up
in
San,
Francisco
and
also
at
mesosphere,
so
I
actually
I
had
to
deal
with
a
lot
of
those
problems
and
you're
also
going
to
see
later,
where
kind
of
connection
comes
back
to
working
on
it
at
a
database
company
and
still
talking
about
machine
learning
infrastructure.
But
let's
actually
dive
into
the
topic.
A
So
this
is
why
I
felt
so
reminded
of
this
book.
Often,
when
working
with
a
Thai,
tea
or
even
being
at
Q
con
talking
about
some
of
the
challenges,
but
the
two
worlds
we
actually
want
to
talk
about
today
is
a
world
of
micro
services
and
the
world
of
machine
learning
models
which
might
seem
very
different
at
first.
But
we're
going
to
see
that's
actually
not
that
different
after
all,
so
Vincent
is
actually
the
expert
from
micro
services.
So
that's
why
I
would
just
hand
over
to
him
now.
Yes,.
B
Thank
you,
so
I
will
start
off
with
this.
This
first
world
micro-services,
assuming
a
bit
that
most
of
you
already
know
debt
and
deploy
services
on
a
daily
basis
in
in
this
world,
so
and
I
think
of
the
layer
over
the
years
the
cease
of
promises
in
challenges
arising
in
this
one
and
especially
towards
deploying
and
releasing
new
services.
First,
we
actually
want
to
make
our
development
teams
independent
and
in
their
own
control
of
driving
their
services
into
the
into
production.
B
We
encourage
them
to
do
more
frequent
deployments,
because
we
think
that
we
can
develop
faster
while
doing
that.
What
are
they
are
there
and
we
don't
want
to
have
more
frequent
incidents
by
doing
that,
so
we
need
some
things
around
that
that
can
actually
help
us
in
in
this,
and
other
stuff
is
actually
we
see
also
that
we
want
to
enable
experimentation
so
that
we
that
we
can
actually
measure
that
the
things
we
are
deploying
into
production
is
actually
yeah.
B
The
things
that
we
expect,
and
so
what
I
want
to
go
through
is
a
couple
of
those
patterns
that
you
that
you
see
in
this
world
and
the
first
one
I
think
is
my
familiar
blue,
green
and
maybe
some
hands
of
people
are
deploying
into
production
with
Bluegreen
deployments.
That's
quite
some.
There
yeah
there,
where
we
push
another
version
of
an
application
next
to
the
current
one
and
make
sure
it's
healthy
and
as
soon
as
it's
healthy,
we
swap
over
the
traffic
to
the
to
the
new
application.
B
B
This
see:
okay,
okay,
good
yeah,
we're
actually
you're
yeah
you're
experimenting
that
the
new
version
should
behave
and
is
indeed
better
than
the
current
version,
and
you
put
it
next
to
that
and
you
look
at
all
the
metrics
that
comes
with
that
and
that's
on
not
only
the
metrics
of
Alf
or,
but
also
he
spawns
the
guys,
but
also
business
metrics,
saying
hey,
I
assume
this.
This
deployment
drives,
my
conversion
are
better
and
that
are
the
things
that
you
can
test
on
and
I.
B
B
You
put
your
code
in,
there
make
sure
it's
healthy
and
that
is
ready
to
receive
traffic
and
releasing
we
saying
releasing
actually
the
act
to
get
the
code
in
the
ends
of
the
customers
and
and
give
you
value,
and
that's
something
that
we
already
started
five
years
ago
at
that
time,
on
the
maysa
marathon
orchestration
and
we're
using
H
a
proxy
for
four
out
into
traffic
and
in
2016
with
the
developments
we
see
around
kubernetes.
We
added
the
community
support
to
this
as
well,
and
now
AQ
Khan
2019.
B
We
also
introducing
SEO
as
part
of
like
the
the
way
of
handling
the
networks.
Are
we
building
on
top
of
sto
at
now
as
well?
And
if
you
look
at
the
FEM
stack,
we
actually
standing
under
the
shoulders
of
giants,
so
we
were
leveraging
the
technology
that
is
available
to
us,
and
that
goes
from
the
three
major
cloud
providers,
but
also
to
the
orchestrators
like
Mesa
Spheeris,
with
DCOs
and
mesas
Marathon
fqb
net
is
the
orchestrator
and
for
the
networking
using
HD,
poxy
and
now
also
sto
and
for
all
the
metric
related
stuff.
A
And
that
actually
brings
us
to
the
next
Roland's.
It's
actually
machine
learning
models
and
those
challenges.
They
might
actually
look
different
at
the
beginning,
so
but
at
cords
actually
the
same.
We
want
to
productionize
our
code.
We
want
to
productionize
our
services,
so
we
now
just
train
the
model
and
we
actually
have
to
get
value
out
of
that.
A
So
we
have
to
deploy
it
to
production,
which
also
is
called
often
serving
in
the
machine
learning
world
and
the
challenges
that
not
only
happens
once
we're
actually
also
going
to
retrain,
because
we
have
received
more
training
data.
Our
data
scientist
have
come
up
with
a
new
cool
model,
so
we
kind
of
have
updated
our
model
and
we
need
to
redeploy
that
and
overtime.
We
also,
we
end
up
actually
with
a
large
number
of
models,
because
we
might
have
different
models
optimized
for
different
environments.
A
One
model
might
work
super
great
on
a
like
a
smart
phone.
Another
might
just
work
really
well
if
we
have
like
a
large
big
server
farm
with,
like
TP
used
or
CPUs,
so
the
challenge
and
how
we
can
actually
make
those
decisions
which
model
or
which
models
we
want
to
deploy,
is
usually
based
on
metadata
and
policies.
And
the
other
thing,
as
you
mentioned,
testing,
like
cannery
deployments,
a
big
challenge
in
machine
learning
is
also
that
testing
is
really
hard.
A
We
cannot
have
like
unit
tests
because
it's
kind
of
like
it's
a
probabilistic
outcome,
for
example
classification.
So
a
really
good
pattern
is
actually
to
just
expose
it
to
live
traffic
and
compare
it
to
others.
So
actually
testing
was
live
production
data,
but
let's
just
step
back.
Why
is
machine
learning
becoming
so
successful
in
recent
years?
We
actually
have
large
amounts
of
data.
A
We
have
large
amounts
of
computer
vailable
and
actually
also
new
techniques
and
like
neural
networks
or
related
machine
learning,
algorithms,
and
that's
actually
turning
our
data
scientists
into
superheroes,
because
they
can
come
up
with
really
cool
use,
cases
from
self-driving
cars,
drug
discovery
and
other
things.
What
do
we
want
our
data
scientists
to
be
doing
they?
A
We
have
to
do
a
lot
of
stuff
around
that,
like
serving
infrastructure,
configuration
data,
cleansing,
data,
preparation,
monitoring
and
so
on,
and
so
on.
So
actually,
this
is
a
lot
of
work
and
many
of
us
are
probably
doing
around
services
as
well,
but
that's
actually
distracting
those
data
scientists
and
what
we
typically
end
up
building
in
the
end
is
like
a
complete
pipeline
from
data
cleaning
towards
the
model
engineering.
So
the
parts
the
data
scientists
actually
graded,
then
model
training,
which
involves
a
large
number
of
resources,
keeping
track
of
all
our
models.
A
We
have
trained,
especially
like
model
training
if
we're
training
a
model.
If
we
train
for
different
hyper
parameters,
this
might
not
result
in
one
model,
but
actually
a
large
number
and
then
actually
getting
production
value
out
of
that
is
in
model
serving
and
actually
model
serving
is
a
part
we're
going
to
talk
most
about
today.
So
one
example
of
that
of
such
pipeline
is
tensorflow
extended.
This
is
kind
of
the
ecosystem,
around
z,
tensorflow
project
and
machine
learning,
and
it's
kind
of
like
matching
what
we
described
before.
A
So
we
have
a
lot
of
data
preparation
part
here,
like
schema
generation,
example
generation
transformations.
Then
we
have
our
trainer
here.
We
have
an
evaluator
which
is
actually
evaluating
how
good
our
metrics
are,
and
then
we
kind
of
deploy
them
and
push
them
into
the
serving
player
down
here,
and
if
we
look
back
so
this
actually
is
this
can't
all
be
managed
by
the
data
scientist.
So
if
we
actually
look
back
how
this
labor
can
be
divided
by
different
superheroes
I
mean
superheroes,
say:
I'll
also
need
sidekicks
to
actually
be
efficient.
A
A
So
basically,
how
can
we
decide
what
models
to
deploy
and
how
to
deploy
models
into
production
and
that's
already
pretty
challenging
by
itself
I,
typically
when
I
work
with
companies
before
this
is
often
an
aspect
which
is
just
forgotten,
because
everyone
is
kind
of
focusing
on
this
data
scientist
spective
Chris.
We
need
just
great
models,
but
actually
the
hard
part
is
later
on
production,
izing
that
model
getting
value
out
of
that.
A
So,
for
example,
the
first
questions
are
like
which
model
to
deploy
if
I
have
a
large
set
of
different
models,
because
even
my
production
data
patterns
might
be
slightly
different
from
the
training
data
I
had
earlier.
So
I
should
keep
metrics
around
and
just
see
whether
I
really
get.
What
I
expected
then
updating
models
without
impacting
those
users
who
might
use
the
hot
dog
Pizza,
app
or
cat
or
doc.
App
I
just
exposed
to
the
Internet,
they
don't
want
any
downtime.
They
want
those
classifications
immediately.
A
So
how
can
I
actually
update
those
models
without
resulting
into
downtime?
How
can
I
deploy
to
different
environments?
As
mentioned?
There
are
different
servers.
There
might
be
different
geographical
regions
so,
for
example,
it's
the
health
care
startup
I
worked
at
it's.
A
lot
of
choices
depends
on
whether
we
use
pH,
I
free,
so
patient
related
data
or
not,
because
that
really
restricts
in
which
environments
you
can
deploy
based
on
data
privacy
issues
and
then,
as
mentioned,
like
testing
models,
it's
also
really
hard
part.
A
I
might
not
really
know
whether
my
model
works
good
with
production
data,
even
though
the
test
accuracy
was
splendid,
it
might
actually
just
fail
in
production
and
another
interesting
pattern
here
is
that
often
we
are
also
not
just
deploying
a
single
machine
learning
model,
but
we
actually
deploy
three
or
four
ends,
and
we
take
like
a
majority
vote
based
on
whatever
is
those
different
models
output,
so
we
kind
of
take
ensemble
decisions,
because
we
actually
get
value
out
of
multiple
different
models.
So
the
question
is
kind
of
like
how
can
we
deploy
that?
A
And
how
can
we
kind
of
aggregate
that
all
together?
And
if
we
actually
look
at
that?
This
is
very
similar
to
the
slide
we
saw
before
about
microservices.
We
want
to
update
a
model.
We
can
do
that
in
like
an
a/b
style,
where
we
have
model
version
1,
and
then
we
have
model
version
2
and
we
just
switch
traffic
over.
A
We
might
want
to
test
models
where
we
slowly
expose
5%
of
the
traffic
to
model
B
and
keep
the
rest
at
the
other
model,
and
we
just
compare
like
how
has
a
different
distribution
patterns
in
the
classifications
or
also,
we
might
do
like
more
calm,
ensemble
or
testing
of
models
where
we
kind
of
shadow
deploy
a
model
and
we
still
route
traffic
to
it.
But
we
kind
of
like
ignores
the
output,
so
imagines
it's
load-balanced.
A
So
this
leads
us
to
this
challenge
of
metadata,
and
this
is
actually
tying
back
to
why
I'm
still
so
interested
in
that
field,
because
this
keeping
track
of
metadata
is
super
challenging,
but
also
super
interesting,
because
it
really
gives
me
value
out
of
this
entire
ml
pipeline.
So
what
we
see
up
here,
this
is
again
like
tensorflow
extended,
so
the
different
components
and
each
of
those
components
actually
is
producing
metadata.
So
we
know,
for
example,
where
our
datasets
are.
We
know
which
transformations
have
been
done
to
our
data
set.
A
We
know
what
training
and
test
accuracy
we
had
when
training
our
model
and
then
also
from
serving
we
might
get
some
statistics
and
it's
actually
really
crucial
that
I
have
like
a
common
metadata
store
where
I
keep
track
of.
All
of
that.
So,
first
of
all,
probably
we
all
think
about
like
accuracy,
so
how
accurate
is
my
model,
but
another
important
factor
to
decide
whether
we
want
to
deploy
it
or
not,
typically
its
latency?
A
How
long
does
it
takes
a
model
to
make
a
decision,
because
that
also
varies
quite
a
lot
between
different
models
and
can
also
even
vary
between
different
environments?
Do
I
have
TP
use
available.
Do
I
only
have
a
smartphone
available?
This
probably
is
going
to
impact
the
choice
of
model
I
want
to
deploy,
and
it's
mentioned,
like
data
privacy,
is
another
interesting
aspect
where
I
just
need
kind
of
like
a
common
view
across
that
pipeline.
A
We
have
a
document
description
of
our
pre-processing
steps
of
our
features
and
so
on
and
so
on.
So
actually
we
have
a
connection
of
unstructured
data
describing
each
of
those
entities
and
then
they
are
just
related
by
edges
by
actually
a
graph.
And
so,
if
we
look
at
that
tree
I
mentioned
earlier,
we
have
to
figure
out
those
models
being
served
in
production
figure
out
all
models
which
have
contributions
from
my
data
says
it's
actually,
it's
a
graph
traversal
find
all
those
models
which
are
kind
of
related
through
those
arches.
A
With
that
model
we
are
using
in
production
there
and
that,
for
example,
enabled
us
to
really
easily
identify
all
those
models,
and
this
is
tying
back
to
why
I
still
care,
as
I,
currently
work
at
a
Rango
DB,
which
is
like
a
multi
model
database.
So
we
actually
can
both
treat
those
document
stories.
It's
unstructured
data,
but
also
evaluates
the
connections
between
them
really
well,
because
it's
actually
this
native
native
model
model.
A
It
actually
allows
me
to
both
have
a
document
store,
plus
a
graph
store
in
like
a
single
database
core,
and
it
also
can
scale
to
multiple
nodes.
So
this
graphs
are
not
restricted
to
a
single
node,
because
often
our
graphs
gets
rather
large
because
a
lot
of
models,
a
lot
of
data
sets
and
so
on,
and
so
on
and
yeah.
This
actually
allowed
us
to
decide
which
models
we
wanted
to
deploy
into
production,
and
this
also
helps
us
to
kind
of
connect,
those
both
worlds
together
soon.
A
B
You
so
whatever
demo
is
actually
how?
How
does
does
one
world
works
for
that
I
want
to
start
off
with
doing
and
canary
release
of
a
service
which
we
have
created
for
this
demo
and
in
this
actually,
and
in
this
demo
we
have
this
this
one.
We
have
this
one
service
running
on
fee
one
and
it
has
in
catalog,
and
the
sheriff
is
that
I'm
showing
is
some
called
smart
Kong.
It's
it's.
B
A
Motivation
was
actually,
if
you
first
time,
I
looked
at
scat
and
especially
like
a
lot
of
newcomers.
I
was
actually
sitting
on
the
plane
next
to
someone
who
was
like
okay,
which
sessions
should
I
go
to
and
I
feel
over
time.
Koukin
has
become
so
large.
That's
actually
a
really
hard
problem,
and
this
is
why
we
were
discussing
like
what
demo
use
case
should
we
take.
It
was
kind
of
like
okay.
How
can
we
recommend
sessions
to
people,
and
that
was
kind
of
the
idea
here?
Yeah.
B
It's
deployed
on
cube
Nettie's
and
we
have
built
a
nice
graph
on
a
desktop
that
show
all
kind
of
information
like
the
incoming
requests,
the
success
rates
and
it's
actually
getting
all
this
data
out
of
sto
we're
constantly
and
over.
Here.
We
also
have
a
custom
metrics
inserted
here,
which
is
the
the
number
favorites
per
minute
and
later
on.
You
can
also
see
the
number
of
favorites
per
model,
so
what
I
will
do
is
actually
starting
canary
release
using
using
femp
and
for
that
I'm
going
into
my
terminal.
B
B
At
the
service-
and
it
will
actually
show
me
that
the
service
is
now
from
version
one
as
a
rate
of
100
and
version
2
as
a
rate
of
zero
and
well,
it's
actually
going
to
slide
those
rates
over
to
the
other
version.
We
actually
made
some
sort
of
error
in
this
this
part.
So
as
in
Canaries
release,
you
do
as
as
long
as
it
doesn't
work.
It
should
fall
back.
B
C
B
The
families
will
actually
give
it
some
actually
give
it
some
farro
use.
We
say
average
services
we
actually
want
to
release
what
is
the
destination.
So
if
you
are
familiar
with,
sto
and
destination
should
sound
familiar
and
we
give
it
to
which
subset
it
has
to
go
and
some
some
parameters
like
in
which
period
and
what
are
the
steps
it
has
to
involve,
and
we
have
baked
in
a
couple
of
release
policies
with
the
event-
and
this
is
in
health,
Dave
canary
release.
Oh
well,
it's
moving
those
traffic
it
constantly
looking
at
a
health.
B
We
also
have
a
time-based
way
of
doing
that.
So
it
goes
regardless
over
time,
but
you
can
also
put
in
your
custom
metrics.
So
as
things
that
are
important
for
you
like
it
resonates
or
it's
Bonus
Time.
If
I
quickly
come
over
here,
we
see
there
is
an
a
problem
in
our
service
and
if
we
look
at
that,
it
automatically
switched
back
to
100
again.
B
B
B
C
B
B
There's
now
section
called
discover
and
if
I
go
there,
you
see
it
has
less
less
sessions
to
show
and
it
will
match
up
towards
my
my
profile
at
least
that's
my
assumption.
Whatever
look
into
into
qivana
I
actually
see
that
model
one
over
here
doesn't
provide
me.
The
expected
amount
of
favorites
per
minute,
so
I'll
see.
Okay,
I
need
to
to
update
this
to
a
better
version
of
my
model
and
for
that
I
actually
going
to
launch
an
experiment.
B
And
this
is
experiment.
Actually
we
treat
the
model
the
same
as
a
service
and
we're
going
to
have
some
traffic
from
from
model
version,
one
to
mold
version
two,
and
while
we're
doing
that,
we're
actually
going
to
look
on
a
what.
What
are
the
metrics?
That
we
expect
for
that
in
this
case,
is
the
favorites
that
one
should
actually
go
go
up
so.
B
B
So
I
started
having
a
50%
of
my
traffic
towards
the
experiment
and
while
doing
that,
I'm
actually
testing.
If
we
getting
more
favorites
out
of
that
model
and
as
we
now
should
show
in
Havana,
this
should
become
more
favorites
into
my
graphs.
So
we
have
to
weigh
down
a
bit
and
what
we
do
with
experiments
is
quite
nicely
we
can.
B
Actually,
if
you
have
a
new
website
and
we
held
traffic
to
us,
the
websites
we
place
them
as
they
come
in
into
an
embark
it
so
we
say,
is
50%
of
the
people
coming
in
to
the
website.
We
place
it
into
the
experiment
and
we're
going
to
monitor
yeah.
If
that
that
part
goes
towards
the
target
that
we
expect
and
we
should.
C
B
Great,
thank
you,
you
saved
it.
As
you
can
see.
Now
we
get
more
favorites
on
the
model,
two
version
of
the
model
and
it
also
increase
our
favorites
per
minute.
So
this
indicates
that
this
version
of
the
model
performs
better
and
better
than
as
we
expect,
and
it
actually
won
the
experiment
and
that's
what
we
don't
then
release
automatically
in
production.
A
B
So
we
can
annotate
the
request
with
which
rolled
the
person
is
in
and
we
can
route
the
traffic
towards
the
model.
That's
actually
connected
to
that.
This
is
a
way
we
can
actually
help
as
well
and
the
other
one
is
actually
using
shadow
traffic
so
meaning
that
we
don't
expose
the
end
user
towards
the
the
new
model.
But
on
the
background,
we
actually
look
at
things
like
performance
and
that
the
outcome
of
the
of
the
results
that
model
gave
us
are
correct
until
we
actually
give
traffic
towards
to
work
it
until.