►
From YouTube: Scaling Open Source ML: How Wolt Uses K8s To Deliver Great Food to Mill... Stephen Batifol & Ed Shee
Description
Don’t miss out! Join us at our upcoming hybrid event: KubeCon + CloudNativeCon North America 2022 from October 24-28 in Detroit (and online!). Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Scaling Open Source ML: How Wolt Uses K8s To Deliver Great Food to Millions - Stephen Batifol, Wolt & Ed Shee, Seldon
Forecasting supply and demand, serving restaurant recommendations and predicting delivery times. These are just a few examples of how Machine Learning is being applied at Wolt. Now with over 12 million users, scaling the ML infrastructure has been a significant challenge. This talk will highlight those challenges and how they were addressed by building an end to end MLOps platform on Kubernetes. You'll learn about the open source frameworks that Wolt integrated, specifically Flyte, MLFlow and Seldon Core.
A
So
yeah,
so
maybe
for
the
people
that
don't
know
walt
we
are.
We
started
as
a
food
delivery
company
in
helsinki
in
2016,
and
we
are
now
in
23
countries
going
from
norway
to
japan,
and
then
we
have
millions
of
users.
That's
like
the
boring
part,
but
the
fun
part
is
that
we
have
machine
learning
and
we
have
a
lot
of
different
use
cases.
The
first
one
is
supply
and
demand
forecasting.
The
second
one
is
recommended
systems.
Then
we
have
logistic
optimization.
A
So
with
a
lot
of
different
machine
learning,
use
cases,
we
have
a
lot
of
different
needs
and
we
have
to
address
those
needs
and
the
first
one,
the
biggest
need
that
you
can
have.
Usually
when
you
train
machine
learning,
is
data
access?
How
do
you
access
your
data?
How
do
you
make
sure
that
you
access
it
in
a
secure
way,
but
it
has
to
be
simple
as
well.
A
We
want
to
have
that
simple
and
yet
secure,
and
then
infrastructure
as
well
has
it
has
been
told
you
know
scaling
is
quite
hard
and
then
you
need
infrastructure
to
scale
usually,
and
when
you
train
a
model,
you
need
some
resources.
You
need
some
computer
resources,
so
you're
gonna
need
cpu,
usually
memory
as
well,
and
then
you
might
also
need
gpus
and
you
might
need
one
gpu
or
five
gpus
or
even
more.
So
how
do
you
have
access
to
those?
A
I'm
a
big
believer
that
if
you
need
something,
it
should
be
easy
to
have
access
to
it.
So
if
you
need
one
gpu,
you
should
just
say:
okay,
I
want
one
gpu.
You
should
not
have
to
think
about
all
the
drivers
and
everything
so
yeah.
We
have
a
whole
infrastructure
for
that
and
then
one
part
that
is
very
important
is
fast
deployment.
A
You
know
you
machine
learning
is
very
iterative
and
if
you
make
it
slow
for
your
data
scientist
to
deploy
something,
then
usually
the
whole
process
is
becoming
very
slow
and
then,
instead
of
taking
three
months
to
deploy
model
to
production,
for
example,
you
take
six
months
a
year
or
even
you
don't
even
deploy
it.
You
know
we
know
like
a
lot
of
models
that
are
you
train
your
model
you're
like
very
happy
with
it,
but
then
you
never
deployed
to
production.
A
So
we
have
data
centers,
the
trainer
model,
they're
very
happy.
They
deploy
it.
But
you
know
I
don't
want
them
to
have
to
go
on
graffana
and
be
like
okay,
I'm
going
to
create
my
new
dashboard
there,
because
most
of
them
they
just
want
to
use
graphana,
they
don't
really
care
and
they
don't
want
to
create
dashboards,
and
also
you
train
your
model,
so
you're
going
to
have
logging
and
we
want
it
to
be
the
same
for
everyone
at
least
for
the
basic
you
know
you
can
have.
A
I
don't
know
like
cpu
usage
memory,
usage
and
all
those
things
they
shouldn't
have
to
think
about
it.
We
should
just
yeah
provide
it
for
them.
So
that's
those
are
the
things
and
the
different
needs
that
we
have
and
yeah.
Maybe
a
quick
reminder
of
the
typical
life
cycle
of
a
machine
learning
say
you
have
your
your
idea.
You
have
a
problem,
usually
so
you're
going
to
collect
some
data
and
then
you're
going
to
create
one
model.
A
Usually
the
first
model
is
going
to
be
terrible,
then
you're
going
to
evaluate
the
model
and
you
create
a
new
model,
and
then
you
collect
more
data
and
everything.
And
then
you
have
a
whole
life
cycle
here
and
then,
after
after
a
bit
of
time,
you're
like
okay,
I'm
pretty
happy
with
my
model
here
it
looks,
looks
decent.
So
what
you're
going
to
do
usually
is
that
you
want
to
deploy
it
to
production,
so
you
deploy
to
production.
A
You
have
your
first
model
deployed
you're
very
happy,
but
then
you
need
to
monitor
it
because
you
know
you
have
a
model,
but
then,
if
you
don't
know
the
latency,
for
example,
of
your
model,
if
you're
doing
it
online
inference,
then
it's
pretty
bad
and
it's
the
same.
You
also
have
to
monitor
the
quality
of
your
model.
A
You
trade
it
on
data
from
the
past.
Usually
so
you
have
now
new
data
that
is
coming,
and
so
you
also
need
to
make
sure
that
you
know
your
model
is
the
same
quality.
It's
like
yeah.
Maybe
you
deploy
something
and
the
data
is
very
different.
Now
that
what
you're
training
off
so
you
really
have
to
be
careful
with
that
and
then
yeah
iterate
again
you
go
back
to
the
problem.
You
go
back
to
collecting
data,
prototyping
models
and
deploying
again
and
that's
like
the
thing
we
really
wanted
to
focus
on
making
iterative
work.
A
Quick
and
easy.
That's
our
main
thing
and
we
had
different
challenges
when
we
started
the
the
whole
ml
platform.
The
first
one
is
that
we
had
projects
that
have
been
running
for
a
long
time
at
world
and
we
we've
been
using
measuring
for
I
think
five
years,
but
we
only
had
a
couple
of
projects
that
were
running
and
it
was
very
hard
to
create
new
projects.
A
We
had
a
lot
of
people,
we
have
some
data
scientists
and
they
have
amazing
ideas,
but
sometimes
it
was
very
hard
to
create
a
new
project
and
be
like
okay.
I
want
to
work
on
that,
but
then
you
don't
have
access
to
the
data
or
then
you
don't
know
exactly
how
to
deploy
something.
So
that
was
quite
hard
and
then
the
second
one
is
that
we
had
a
lot
of
different
tools.
A
We
had
we're
using
airflow,
then
we're
using
also
some
cron
job
to
train
your
model,
but
also
yeah,
some
save
maker
and
then
sometimes
some
custom
code.
You
know
like
someone
wrote
three
years
ago
and
then
no
one
knows
exactly
what
it's
doing.
So
that's
that
was
like
a
big
problem
that
we
had
and
the
last
one
is
really
the
impact,
because
we
didn't
know
exactly
what
was
running
or
if
it
was
running
well,
it's
also
very
hard
to
monitor
the
impact
of
your
models
and
be
like
okay.
My
model
is
pretty
good.
A
You
know
when
I
order
food,
it
arrives
on
time
or
not,
and
so
that's
those
are
models
that
are
very
important
for
us
and
we
couldn't
really
monitor
it
and
one
thing
that
usually
people
don't
really
think
about
is
like
we
had
a
disconnect
between
match
learning,
metrics
and
business
metrics.
So
you
arrive
to
a
meeting
you're
a
data
scientist
you're
very
happy
your
mean
squad.
Error,
for
example,
is
lower,
but
then
we
didn't
exactly
know
what
it
was.
A
What
does
it
mean
for
the
business,
and
so
it
was
like
a
big
big
struggle
to
be
like
okay.
So
when
you
start
a
new
project,
you
have
some
metrics
that
are
data
metrics,
and
then
you
have
some
business
metrics.
So
you
want
to
optimize
for
both
and
then
when
you
have
meetings,
then
it's
a
bit
easier
to
be
like
okay,
my
my
error
is
lower.
A
Therefore,
the
the
business
should
increase
or
the
retention
should
increase,
and
so
those
were
the
main
challenges
that
we
had
and
if
our
clicker
works
yes,
so
what
do
we
want
exactly
for
the
machine
learning
platform?
Well,
the
first
one
is
easy
to
launch
and
iterate.
I
said
that
already
a
couple
of
times,
but
that's
the
main
thing
and
then
really
a
driving
force
for
new
endeavors.
You
know,
you're,
like
you,
create
a
new
project
and
you're
like
yeah.
We
have
a
whole
ml
platform
and
it's
pretty
easy
to
train
a
model.
A
It's
pretty
easy
to
deploy
your
model,
and
then
we
also
we
have
monitoring.
We
have
logging,
so
you
know
we
can.
We
can
create
it
and
we're
pretty
confident
that
we
can
make
it
an
end-to-end
model
actually
and
focus
on
platform
velocity,
that's
a
big
thing
as
well
and
then
tooling,
so
common
best
practices.
A
You
know
I
have
like.
We
have
a
lot
of
tests.
We
write
a
lot
of
tests
for
data
scientists
and
then
they
can
use
them,
but
then
we
also
have
different
tooling,
like
we
have
an
in-house
tool
that
is
doing
yaml
to
terraform,
for
example,
so
our
data
scientist
when
they
want
to
access
some
bucket
on
s3.
Instead
of
writing
all
the
terraform
and
everything,
then
we
have.
We
write
the
yamufl,
the
yamo
with
the
s3
bucket
and
then
it
will
create
the
whole
terraform
without
you
having
to
do
anything.
A
A
So
you
you
can
train
your
new
model
and
with
the
different
tool
that
we
use
and
I'm
going
to
talk
about
it
later.
You
need
to
write
a
docker
file.
If
you
want
to
train
your
model,
and
for
that
you
know
you,
you
might
not
want
to
run
your
docker
file
as
root
or
you
know,
different
things
that
are
related
to
security.
A
So
we
have
those
templates
so
that
people
can
just
use
them
and
then
they
replace
they
replace
the
template
with
their
code.
But
then
most
of
their
work
is
already
done,
so
they
don't
really
have
to
think
about
those
things
and
therefore
it's
deployed
in
a
secure
way.
I
can
say,
and
also
a
lot
of
automation,
that's
something
very
important.
A
You
work
on
your
feature
branch.
You
merge
it
to
the
main
branch
and
then
we
have
a
lot
of
ci
cd
running.
We
have
automatic
deployment
as
well
running
if
you
image,
but
I'll
come
back
to
that
a
bit
later
and
the
impact
yeah.
We
want
logging
continuous
monitoring
by
default
for
everything
so
for
the
model
that
you're
training
for
the
model
that
you're
deploying-
and
we
want
that
by
default,
for
everything
without
having
the
data
sensors
to
do
anything
and
yeah
make
machine
learning
a
core
business
component.
A
So
yeah,
and
how
did
we
start
then
started
to
create
value
very
quickly
for
a
data
scientist,
our
data
scientist,
our
customers?
I
guess
most
of
you
work
with
customers.
Usually
you
have
to
create
some
values
for
them,
so
they're
happy.
But
it's
the
same
for
us.
We
had
to
create
values
for
them
quickly,
so
that
means
yeah
first
iteration
within
automatic
monitoring
but,
for
example,
being
able
to
run
a
model
in
shadow
mode.
You
have
a
model
running
in
production.
A
A
Maybe
if
I,
if
it's,
if
it's
broken
sorry,
then
you
know
you're
not
going
to
update
everything
and
then
you
have
the
whole.
What
communities
allows
you
to
to
have
with
running
updates
and
it
had
to
be
easy
to
deploy
on
kubernetes
as
well,
because
when
I
joined
the
company
I
joined
a
year
and
a
half
ago,
you
know
we
already
had
like
14
data
centers,
and
I
was
the
only
one.
So
you
did
you.
I
had
to
earn
trust
of
the
data
scientists
and
the
different
stakeholders.
A
You
know
I
didn't
want
to
be
to
arrive
there
and
be
like
okay.
Now.
Everyone
trust
me,
you
know,
have
no
background
in
building
an
ml
platform,
so
I
had
to
earn
their
trust
and
to
do
that,
we
had
to
deploy
something
that
was
quick
to
deploy
on
communities
and
the
last
one.
We
have
a
motto
at
walt,
which
is
to
focus,
and
so
for
that
we
also
wanted
to
focus
on
only
one
component,
meaning
that
we
didn't
want
to
build
the
whole
ml
platform.
A
You
know
from
the
first
month
we
had
to
only
one
component
that
creates
value
quickly
and
we
can
have
a
quick
impact
so
that
we
don't
spend
you
know.
I
don't
know
six
months,
eight
months,
working
on
something
something
that
can
be
amazing,
but
then
your
data
scientists
will
be
okay,
I'm
not
gonna
use
it
because
I
don't
need
it.
You
know
we
want
quick
feedback
as
well
from
them.
We
expect
quick
iteration
from
them,
but
in
return
we
want
quick
feedback
from
them,
so
you
can
be
like
okay.
What
you
did
is
pretty
cool.
A
A
So
we
have
a
whole
ml
platform
now
and
we
have
flights
here
which,
which
is
what
we
use
to
train
and
orchestrate
our
different
workflows.
So
that's
running
on
kubernetes
as
well,
and
it's
fully
distributed.
It's
fully
scalable
and
it's
running
on
dedicated
kubernetes
clusters
that
we
have,
and
then
we
use
ml
flow
for
experiment
tracking.
A
Well,
it's
very
famous
but
yeah.
You
can
track
metrics
and
different
things
related
to
your
models,
and
then
we
have
some
python
services
that
are
here
to
provide
automatic
updating
for
your
model
and
different
things
that
we
need
and
the
one
that
I
mentioned
here
as
well,
which
is
the
yamal
to
terraform.
It's
also
something
that
we
have
here
and
the
last
one
it's
seldom
core.
A
So
it's
what
we
started
with
it's
our
deployment
service
and
it
basically
allows
you
to
put
models
into
production
microservices
and
to
have
automatic
logging,
automatic
monitoring,
shadow
mode,
a
b
test,
and
so
that's
what
we
have
now
and
that's
what
we're
going
to
build
on
in
the
future.
So
yeah
go.
B
B
B
You
know
and
you'll
see
some
of
that
again
when
I
show
that
in
a
second,
it
has
its
own
model
format,
which
is
really
good
for
making
your
models
particularly
different
frameworks,
reusable
in
the
same
way,
and
then
it
also
has
a
model
registry
which
is
a
way
of
handling
the
life
cycle
of
a
you
know:
machine
learning
model,
particularly
if
you're
using
mlflow
to
track.
You
know
the
training
runs
and
stuff
like
that.
B
If
we
dig
a
little
bit
deeper,
what
it
actually
does
is
it
creates
these
containerized
microservices
with
a
rest
and
grpc
interface
and
then,
if
we're
digging
in
even
further
right,
it
actually
provides
these
like
highly
optimized
inference
servers
that
you
can
then
use
to
to
execute
your
model's
code
on
when,
at
you
know,
at
runtime
when
you're
you're
doing
inference,
and
if
you
have
a
look
on
the
right
there.
That's
an
example
of
a
very
simple
selden
deployment.
B
Crd
and
they-
I
don't
know
if
this
point
is
gonna
work,
but
you
can
see.
For
example,
I've
got
like
this
implementation
key
value
pair
and
then
I've
got
you
know,
sklearn
server,
so
it's
using
the
scikit-learn
server
and
I've,
given
it
this
model
uri.
So
I
can
go
and
pull
my
model
down
and
deploy
it
on
a
on
an
optimized,
scikit-learn
server
cool.
One
of
the
things
I
really
really
like
about
solon
as
well
is:
is
this
ability
to
do
complex
inference
graphs?
B
So
we
appreciate
that,
like
not,
every
model
just
runs
on
its
own
standalone.
Often
you
know
you
have
input
transformers
output
transformers,
you
might
have.
You
might
want
to
chain
models
together.
I've
given
a
kind
of
example
up
here
of
how
you
might
do
maybe
two
models
running
in
tandem.
If
I
kind
of
think
of
an
example
for
this,
maybe
you
know
that
if
you
look
down
the
left-hand
side
here,
this
model
two
might
be
a.
You
know
some
sort
of
like
image
classification,
where
I'm
doing
some.
B
You
know
transformation
on
on
it
beforehand
and
then
some
output,
particularly
nlp.
We
see
a
lot
of
use
cases
for
that.
You
know
converting
to
word,
embeddings
running
a
model
and
then
you
know
converting
back
to
some
sort
of
text
on
the
on
the
output
and
then
maybe
model.
One
here
just
takes
standard
tabular
data
as
in
numerical
inputs,
and
then
you
can
have
things
like
combiners
right
which
take
the
outputs
of
multiple
models
and
then
feed
those
back.
B
So
it
allows
you
to
define
the
whole
machine
learning
like
almost
business
logic
within
your
deployment,
which
means
your
end.
Users
only
have
to
hit
one
api
and
they
get
all
the
results.
They're
expecting
and
there's.
I've
got
a
kind
of
example
of
how
that
you'd
build
that
up
within
the
crd
a
couple
other
things,
so
it
integrates
really
nicely
with
everything
in
the
kubernetes
stack.
You
know
we
use
a
lot,
for
you
know:
request
logging
response,
logging,
etcetera,
we
use
elasticsearch
for
that,
but
you
you
could
plug
and
play
different
databases.
B
You
can
do
distributed
tracing
with
jaeger.
It
automatically
will
scrape
metrics
from
your
models
and
and
the
containers
underneath
pass
them
to
a
prometheus
server.
We
do
stream
processing
integrations
with
k
native
and
kafka,
and
then
you
can
integrate
with
batch
workflow
managers
like
argo,
or
we
actually
have
a
cli
tool
that
allows
you
to
kind
of
run
batch
processing,
workloads
too.
Okay.
B
So
that's
enough
talking,
I'm
going
to
show
you
something
a
bit
of
a
demo,
I'm
going
to
have
to
squat
a
little
bit
here
because
I'm
too
tall
for
this
lectern,
so
maybe
maybe
I'll
go
for
a
power
stance.
Okay,
just
before
I
start
I'll
talk
about
the
the
use
case,
so
obviously
I
can't
give
you
any
of
walt's
actual
machine
learning
models
and
their
their
data
and
kind
of
show
it
live.
B
So
I've
tried
to
pick
something
that's
kind
of
similar,
so
we're
gonna
do
some
food
classification
with
tensorflow
they're,
basically,
a
bunch
of
images
that
have
been
taken
at
different
restaurants
by
people's
phones
and
so
on.
So
you'll
see
that
the
quality
varies
quite
a
lot,
and
maybe
the
motivation
for
doing
this
is
that
if
you
can
classify
every
image
that
you're
using
on
you
know
like
a
food
ordering
app
right
and
have
the
metadata
attached
to
it,
you
can
then
build
up
a
profile
of
a
user.
B
You
know,
as
I
click
on
certain
images
and
go.
Oh
yeah
that
looks
nice
right
et
cetera.
I
can
be
storing
that
metadata
and
use
it
for
recommender
systems
later
on.
So
let's
dive
into
that
right
and
I'll
see.
Can
you
can
someone
at
the
back?
Tell
me
if
you
can
see
this
all
right.
Do
I
need
to
make
a
bit
bigger?
Okay,
let's
I
don't
know
what
this
means
thumbs
up.
I
think
big,
bigger,
okay,
cool
right.
Let's
go
bigger
all
right,
fine,
so
I'm
gonna.
B
Do
I'm
just
gonna
get
something
training
first,
so
I've
got
this
again
I'll
make
this
one
bigger
too.
Presumably,
that's,
okay,
okay,
great!
This
is
again
a
simple
kind
of
convolutional
neural
net.
The
things
to
highlight
in
this
code,
rather
than
going
through
it
in
detail,
is
this
line.
Here
is
really
cool
right,
so
if
you
import
ml
flow,
you
can
you
can
actually
just
run
one
line
of
code
there,
this
mlflow
dot
keras
to
auto
log.
B
It
has
this
auto
log
feature
for
a
bunch
of
frameworks
and
that
will
then
automatically
track
a
bunch
of
parameters
metrics
around
your
model
and
then
store
your
model
into
one
of
those
reusable
ml
flow
model
formats
as
well.
So
that's
enough
right
to
do
kind
of
80
of
what
you
want
to
do.
You
can
also
do
things
like
you
know,
log,
maybe
manually
log,
some
parameters
in
there,
so
I'm
going
to
do
batch,
size,
epochs
drop
out
et
cetera,
and
then
what
I'm
going
to
do
actually
is
I'll.
B
Okay,
and
that
might
that
I
think
how
many
epochs
did.
I
do
only
three
so
we're
good.
That
should
take
a
little
bit
of
time.
Okay,
and
you
can
see
the
kind
of
examples
of
images
that
are
being
classified
by
this.
You
know
some
are
good.
Some
are
a
little
bit
more
ropey
right.
You
know
down
here
this
pizza
image.
I
mean
that
kind
of
could
be
anything.
B
Maybe
if
you
look
closely,
you
can
see
it's
it's
a
pizza
cool,
so
we'll
let
that
run
and
while
we
do
I'm
going
to
jump
in
and
show
you
the
ml
flow
ui.
So
if
I
refresh
this,
you
should
see
that
hang
on.
Any
second
is
that
running
cool
yeah.
It's
already
picked
up
my
new
training
run,
even
though
it
hasn't
finished.
B
So
if
I
jump
back
down,
you
can
see
it's
still
training,
but
it's
picked
up
some
of
those
parameters
I
logged
already
and
then,
if
I
jump
in
and
show
you
something
you
know
from
a
previous
run,
the
cool
thing
here
is,
I
can
look
at
like
I
can
compare
metrics
across
you
know
over
time.
All
of
this
is
automatically
logged
for
me
by
ml
flow
tracking.
So
if
I
want
to
look
at,
maybe
you
know
the
accuracy
versus
the
validation
accuracy
we
can
see
for
this
model.
B
Let's
put
this
for
epochs
right
rather
than
time,
maybe
after
about
30
epochs,
I'm
just
overfitting,
because
you
can
see
the
the
accuracy
of
the
model's
going
up,
but
the
validation
accuracy
is
not
so
that
maybe
you
know
that's
a
good
indicator
that
I
probably
don't
need
to
train
for
that
long.
But
there's
maybe
some
data
enhancements
I
need
to
do.
I
can
also
do
cool
things
like
compare
runs.
So
if
I
go
in
and
you
know
grab
two
of
them-
let's
go.
B
Compare
those
and
then
again
you
know,
look
at
the
different
accuracy
or
loss
metrics
in
there.
You
can
see
so
this
one
that
had
a
larger
batch
size
as
we'd
expect
right
kind
of
converges
on
a
reasonable
solution,
much
quicker
because
the
machine
learning
model
has
more
information
about
to
read
with
each
each
batch
cool.
So
let's
jump
back
down
here,
we'll
see
that
my
model's
completed
and
actually
wait
just
quickly
before
I
do.
I'm
gonna
jump
in
here
and
show
you
that
you
know
again,
you
get
all
the
same
information
for
that
run.
B
B
So
what
that's
going
to
do
is
just
saves
me
a
bit
of
navigating
around
the
file
system,
but
that's
copying
that
saved
mlflow
model
into
a
mineo
storage
bucket
on
my
cube
cluster
and
then
from
putting
it
there.
What
that
means
is.
I
can
now
create
a
seldom
deployment
and
deploy
that
straight
on
top
of
kubernetes.
B
So
let's
have
a
look
at
what
one
of
those
looks
like
right.
So,
let's
go
hang
on
I'll
play
in
a
second
okay,
so
this
is,
for
the
sake
of
this
demo
right
there's
one
already
running.
This
is
my
model.
That's
already
up
and
running
my
food
classifier.
That
already
can
accept
predictions,
and
you
can
see
in
here.
You
know
some
of
the
obvious
things
I
pointed
out
earlier,
like
I'm
using
this
ml
flow
server,
because
it's
using
that
ml
flow
model
format
and
then
I've
passed
it.
B
This
model
uri,
which
basically
is
saying
like
where,
where
is
my
model
stored,
you
know,
go
and
grab
that
and
then
importantly
here
as
well
like
because
that's
in
an
s3
bucket
or
in
this
case
on
mineo
running
on
my
cluster,
how
do
I
actually
access
that?
So
I
have
to
pass
this
secret
in
as
well,
but
that's
basically
it
in
terms
of
what
you
need
and
then
you
can
see
actually
like
in
one
of
the
cool
things
with
seldom
core.
B
Is
you
can
customize
like
a
whole
load
of
the
crd
underneath?
So
you
know,
if
you
want
to
write
the
full
container
spec,
you
can
do
that
and
in
this
case
I
actually
needed
to
because
what
ml
flow
does
when
you
deploy
an
mlf
model?
Is
it
recreates
the
whole
environment
from
scratch?
So
for
this
model
it's
going
to
go
in
fact
I'll
show
you
it
in
the
ui.
So,
like
the
model
we
just
trained,
you
can
see
these
artifacts
and
there's
this
like
ml
model
file
and
there's
a
condo.yaml.
B
It
uses
this
with
all
my
python
dependencies
to
recreate
the
actual
environment
that
the
model
is
going
to
run.
In
the
reason
I
then
need
to
set
timeouts
on
my
liveness
probe,
and
my
readiness
probe
is
that
that
can
take
a
while,
particularly
on
you
know.
I've
got
rubbish
wi-fi
here
in
this
venue
and
it's
installing
tensorflow
from
scratch.
Right,
that's
a
pretty
big
download,
so
I
can
do
cool
things
like
that.
What
I'm
going
to
do
for
this
one
is
I'm
just
going
to
go.
B
Show
you
how
maybe
for
a
new
one
that
we've
trained,
which
was
only
three
epochs
and
probably
pretty
rubbish
model,
let's
not
roll
it
straight
out
into
production.
Let's
do
a
shadow
deployment
instead.
So
if
you
look
at
this
one
again,
it's
the
same
as
the
deployment
we
had
earlier,
but
I've
added
another
component
spec
down
here
with
the
same
kind
of
liveness
probe
thresholds,
and
then
this
time
I'm
going
to
let's
put
in
that
new
run
id.
B
So
this
is
the
new
model
we
just
created
and
just
pushed
to
a
bucket,
and
then
all
I
need
to
do
to
specify
that
it's
a
shadow
deployment
is
just
this
one
little
line
here.
So
I
can
do
shadow
true
right
and
now,
what's
going
to
happen
is
all
the
traffic
that
goes
to
the
main
model
is
going
to
be
replicated
to
the
other
one,
but
then
the
output's
just
going
to
be
discarded.
B
So
if
I'm
monitoring
it
with,
you
know
prometheus
and
looking
at
latency
things
like
that,
I
can
make
sure
the
model
performs
well
before
I
actually
promote
it.
I
could
equally
you
know
if
I
wanted,
I
could
do
really
cool
things.
Like
just
say,
you
know
traffic
20
right
and
then
you
know,
go
and
change
the
traffic
up
here
as
well
to
be
traffic
80
right,
and
so
I
can.
B
I
can
do
kind
of
a
b
tests
and
things
like
that
on
my
model
too,
let
me
just
just
undo
that
okay,
so
shadow,
true,
okay,
fine
and
then
how
do
I?
How
do
I
actually
deploy
that
model?
So
that's
just
a
simple
cube:
ctl
apply,
deploy
shadow
right
and
now,
if
we
look
at
my,
you
can
see
that
it's
creating
so
my
food
classifier
defaults
already
there,
but
it's
creating
this
shadow
deployment
as
well,
which,
when
it
receives
requests,
is
going
to
send
to
you
as
well.
B
B
Hopefully
it's
going
to
get
this
right,
but
let's
copy
the
image
url
and
then
what
we're
going
to
do
is
go
predict
and
I'm
just
going
to
give
it
the
url
and
it's
going
to
pull
that
image
down.
Oh,
maybe
not
what
have
I
done
wrong?
Oh
forbidden
it
that
url
doesn't
like
me:
okay,
let's
try
another
one.
B
B
The
reason
I
want
to
show
this
is
because
the
output
of
the
classifier
is
kind
of
cool
you
get
to
see
like
live
inference
requests.
Oh
no.
I've
broken
something
now!
Oh
my!
That's
all!
Okay
nevermind!
I
won't
fix
it
now
because
we
don't
have
time,
but
the
issue
there
is
that
I
haven't
pulled
forwarded
from
my
cluster
to
my
device.
B
A
Thank
you
so
yeah.
Maybe
the
very
important
question
on
that
site
is:
how
exactly
do
we
scale
the
ml
platform,
because
that
was
you
know
the
title
of
the
talk
and
on
our
side
yeah.
So
our
data
scientist,
our
customers
and
what
we
have
now
is.
We
have
dedicated
communities
clusters
only
for
machine
learning.
We
have
ones
running
in
the
development
environment,
production
environments
and
then
we
have
other
ones
as
well.
So
then
you
work
on
your
machine
learning
and
you
use
a
lot
of
a
lot
of
different
data,
a
lot
of
different
nodes.
A
Well,
then,
you
want
to
impact
the
the
rest
of
the
production
data,
so
that's
pretty
good
yeah,
common,
tooling
and
infrastructure
as
well.
As
I
said,
our
yamal
to
terraform,
for
example,
is
one
of
the
best
examples
we
have
automation,
yeah
you
automatically
deploy
to
production
automatically,
deploy
to
shadow
mode
as
well
automatically
do
a
lot
of
things,
a
lot
of
automation
and
the
last
one
is
really
predicting
the
future
needs
of
data
scientists.
A
At
the
moment
we
only
have
one
model
that
is
using
gpus,
for
example,
and
but
maybe
in
a
year
we'll
have.
I
don't
know
five
five
different
models
and
so
they're
gonna
use
gpus
to
train
the
model.
But
then
how
do
you?
How
do
you
run
inference
with
gpus?
Do
you
need
gpus
to
run
inference
or
do
you
use
cpus?
That's
like
one
thing
that
you
try
to
predict
so
then,
when
they
arrive
and
then
a
year
later,
they're
like
okay,
we
need
that
we're
like
okay,
we
try
it
out.
A
You
know
you
should
do
it
like
that.
So
that's
something
that
we
really
try
to
have,
and
I
clearly
don't
have
time
to
talk
about
that,
but
that's
a
very
big
model.
We
had
running
and
we're
totally
blind
on
it,
but
what
we
did
now
is
that
it's
running
on
the
infra
and
because
of
that
now
we
can
see
like
number
of
requests
per
second.
It
receives
and
the
latency
the
memory
usage.
A
But
one
thing
that
is
very
cool
is
like
we
use
kafka
a
lot
and
now
every
response
you
can,
you
can
log
it
to
kafka
automatically.
So
then
you
can
monitor
the
quality
of
your
model
and
then
you
can.
We
use
snowflake
behind
the
scene.
So
then
you
go
to
snowflake
and
you
can
compare
your
prediction
to
the
real
values
and
you
can
do
that
easily,
because
everything
is
locked
to
kafka
and
yeah.
You
can
run
shadow
mode,
a
b
test
and
different
things.
A
So
that's
that's
what
we
have
now
and
yeah
future
work.
I
will
finish
with
that.
We
won't
really
want
to
work
on
the
ergonomics
of
the
platform
data
scientists
for
now
still
have
to
struggle
with
cube
ctl
and
everything.
In
some
cases
we
want
to
get
rid
of
that.
So
we
want
to
make
communities
completely
yeah.
We
want
to
hide
it
from
them.
A
If
they're
not
interested
and
really
have
an
integration
with
the
rest
of
our
tools,
we
have
an
experimentation
platform,
so
we
want
to
have
full
integration
with
that,
so
we
can
run
a
b
test
and
then
monitor
the
result
and
everything-
and
I
will
finish
on
that-
because
we're
out
of
time
so
yeah.
Thank
you.
Cool.
C
D
E
B
You
want
to
yeah
yeah
it.
It
pulls
that
model
down
right
and-
and
that's
something
that's
like
mandatory
within
seldom
core.
Is
that,
like
you,
have
to
provide
an
external
uri
to
the
model
because,
like
if
your
model's
huge
right,
you
don't
want
to
be
like
pushing
that
up
with
a
you
know,
cube
manifest
right
when
you
do
it
so,
and
we
have
an
init
container
that
goes
and
grabs
that
down
pulls
it
down,
make
sure
you've
got
all
the
artifacts
before
it
starts
running
the
classifier.
B
F
A
Yeah,
so
when
I
started,
we
benched
smart,
cube
flow
and
the
whole
thing,
but
I
found
it
to
be
quite
hard
to
use,
especially
for
data
scientists.
You
know
having
to
write
all
the
pipelines
and
all
the
dsl
and
everything
related
to
that.
I
find
it
to
be
quite
hard
even
for
me,
and
I
was
interested
in
the
tool,
so
I
didn't
really
wanted
to
sell
it
to
our
data
scientist,
because
I
think
it
would
have
been
too
completely
complicated
for
them.
So
yeah,
that's
why
we
didn't
go
with
it.