►
From YouTube: Moving People and Products with Machine Learning on Kubeflow - Jeremy Lewi, Google & Willem Pienaar
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Moving People and Products with Machine Learning on Kubeflow - Jeremy Lewi, Google & Willem Pienaar, GO-JEK
GOJEK, the Southeast Asian super app, has experienced a massive increase in customer growth by leveraging machine learning throughout its products. However, building production grade machine learning systems is still fraught with challenges and delays. In order to address these challenges GOJEK has adopted Kubeflow, a fully open source cloud native platform for machine learning. In this talk you will learn how Kubeflow empowers teams at GOJEK by increasing the speed at which they turn ideas into products. One of these products, the GOJEK driver (blog post: http://bit.ly/2MhujA1) allocation system, has been built from the ground up on Kubeflow, and fully utilizes cloud native tooling. We will show how Kubeflow, which leverages Kubernetes and Istio, simplifies the process of rapidly training, deploying, and evaluating models on this multi-objective ML system.
https://sched.co/MPac
A
A
They
form
one
super
app
and
the
goal
of
this
app
is
to
have
a
solution
for
every
working
birthday.
Need
that
you
have
just
to
give
you
an
indication
of
the
kind
of
scale.
So
we
originally
started
in
Indonesia.
But
since
then,
we've
branched
out
into
many
different
markets,
Singapore
Vietnam
Thailand
in
one
of
our
products,
the
right
heading
product
we
have
about
2
million
drivers
on
the
platform
and
we
process
about
a
hundred
million
bookings
every
month
for
more
than
a
hundred
million
bookings.
A
Every
month
in
go
food
were
one
of
the
largest
food
delivery
services
in
Southeast
Asia,
excluding
China.
One
of
the
largest
in
Asia,
with
over
400,000
merchants
on
our
platform
and
then
third,
the
curb
paid
the
product
we
have
is
also
one
of
the
fastest
growing
digital
wallets.
So
the
scale
is
quite
large
and
the
ability
for
us
to
influence
the
business
through
machine
learning
and
data
science
systems
is
quite
immense
and
at
the
heart
of
that
is
data
and
I
hope
this
renders
correctly.
But
this
is
an
animation
of
one
of
our
services.
A
But
when
we
started
we
didn't
really
have
a
platform
or
anything
to
build
on
top
of
so
as
a
data
science
back
firmly,
we
were
tossed
worth
or
at
a
sense
platform
team
we
were
tossed
with,
but
deciding
of
what
what
to
build
our
platform
on
right.
Do
you
use
a
managed
service?
Do
you
both
from
scratch
to
use
something
that
abstract
that
away
like
you
burn
a
DS,
so
we
looked
at
the
problem,
space
and
opportunities
in
the
company,
and
what
we
saw
was
train
like
this,
where
there
was
two
types
of
problems.
A
Basically,
the
first
was
you
had
opportunities
to
put
the
platform
to
effect
model
creation
for
data
scientists
that
work
ephemeral,
wolf
line.
Maybe
they
were
just
going
to
be
used
for
analysis
or
a
small
project
that
never
gets
into
production.
So
that's
on
the
left
hand
side
over
here,
but
there
are
also
these
high-impact
projects
that
we
recorded
sessions
systems.
Essentially
so
a
pricing
engine
or
driver
allocation
or
the
food
recommendations
on
the
homepage
or
any
kind
of
search
engine
within
the
application.
A
So,
from
the
perspective
of
a
platform
that
we
had
to
both
we
had
to
satisfy
both
of
these
orthogonal
objectives.
One
is
provide
an
abstraction
for
the
users
that
makes
it
easy
for
them
to
quickly
iterate
in
bulk
models,
and
the
other
one
is
that
the
platform
should
be
able
to
integrate
with
our
production
engineering
systems,
and
it
should
cater
for
all
the
requirements
of
those
systems
and
handle
the
scale
and
and
be
extensible
and
flexible.
A
So,
given
these
two
orthogonal
constraints,
we
decided
to
focus
on
the
most
impactful
use
cases.
At
the
start,
we
build
our
platform
based
on
one
specific
project
and
and
then
take
it
from
there
and
add
incrementally,
add
more
systems
and
projects
to
it,
and
so
the
one
we
decided
to
start
with
was
the
driver
allocation
in
each
case.
A
So
this
is
a
system
that
takes
a
list
of
drivers
and
the
customer
and
then
decides
which
driver
to
same
to
that
customer
and
the
volume
of
transactions
that
previous
processed
by
one
of
these
smooth
of
this
system
itself
is
very
high,
so
hundreds
of
more
than
a
hundred
million
bookings
every
month.
So
it
has
a
very
high
ability
to
affect
the
bottom
line
of
the
business
and
based
on
the
requirements
and
our
knowledge
of
the
complexity
of
the
system
and
the
importance
of
the
system.
We
need
to
be
engineering
heavy,
so
we
just.
A
We
opted
based
on
our
understanding
of
the
m/l
landscape
to
build
our
platform
on
kubernetes
and
originally
it
was
mostly
because
of
the
dependence,
dependency
management
aspects
of
it,
because
our
data
science
were
already
using
docker
for
their
containers
and
packaging
up
the
dependencies.
But
we
also
got
benefits
like
container
orchestration
and
process
isolation.
So
basically,
what
the
system
did
was
we'd
use
a
very
basic
airflow
pipeline
to
trainer
model
and
deploy
that
model
in
a
web
service
to
container
running
innkeeper
Nettie's.
A
It
would
get
in
the
request,
which
is
a
list
of
drivers
in
the
customer.
Maybe
we
didn't
reach
that
with
some
data
and
then
it
would
give
a
response
on
which
drivers
who
same
to
the
customer-
and
this
had
an
incremental
game
and
it
did
to
improve
our
bottom
line.
But
it
wasn't
that
spectacular
and
one
of
the
concerns
was
the
learning
curve
of
cube,
and
so
we
were
always
questioning
at
the
start
was
going
for
cube
the
right
decision,
but
over
time
it
proved
to
be
indeed
the
right
decision
so
immediately.
A
It
started
paying
dividends
when
we
needed
to
add
experimentation.
So
when
we
started
deploying
multiple
models
into
our
kubernetes
cluster,
we
needed
a
way
to
manage
traffic
to
these
models.
So
we
built
a
very
rudimentary
you
experimentation
user
interface
for
our
data
scientist
so
that
they
can
configure
traffic
splits
to
these
models,
and
one
of
the
benefits
of
kubernetes
was
that
it
provided
a
extended
eyes
control
plane
for
us
to
bowl
on
top
off.
So
we
could
loosely
couple
our
ingress
and
our
traffic
manager,
which
is
in
the
cluster
to
configuration
user
interface.
A
So
so
really,
this
is
one
of
the
key
value
adds
that
kubernetes
brings
when
you're
building
a
platform
on
top
of
cuber
natives.
The
second
value
add,
was
on
orchestration,
so
already
here
we're
seeing
that
the
system
is
getting
a
little
bit
more
complex.
So
what
we
found
was
that
it's
very
rare
to
actually
have
just
a
model
in
a
web
service
serving
these
responses
and
actually
doing
a
good
job.
A
What
we
found
is
that
you
actually
need
many
different
models:
ensemble
together
to
have
a
valuable
output,
and
so
we
bought
an
Orchestrator
called
lasso,
and
if
you
want
to
look
for
an
alternative
and
open-source
well,
there
sell
them
that
you
can
also
look
at.
But
basically
you
define
a
workflow
or
a
graph,
and
this
graph
can
do
things
like
call
multiple
models
and
take
the
most
confident
response
or
pull
two
models,
and
maybe
one
is
fast
and
accurate.
But
one
is
not
it's.
A
Maybe
sorry,
one
is
slow
and
I,
it's
known,
accurate,
but
other
ones
fast
and
not
accurate.
So
if
the
one
is
taking
too
long
to
respond,
then
take
the
other
one
now,
so
you
can
do
all
of
these
complex
logic,
and
that
was
really
a
big
value.
Add
and
that
dramatically
improve
the
impact
of
the
system.
And
then
it
at
the
end
of
the
day,
are
also
experimenting
on
multiple
versions
of
these.
So
Cubans
helped
us
here,
because
you
have
an
orca
sex,
acute
environment
that
shared
between
all
of
these
services
so
running.
A
A
Then
there
were
some
other
benefits
to
running
on
Cube
and
at
just
economies
of
scale.
So
we
also
started
to
standardize
across
the
company
on
kubernetes.
It
wasn't
just
the
data
science
of
the
ml
teams
and
a
lot
of
the
benefits
were
being
felt
by
our
team
by
what
the
other
teams
were
building.
So
in
one
case,
we
wanted
to
log
the
incoming
requests
to
our
services,
but
because
of
the
performance
constraints,
there
is
actually
too
hard
to
do
this
ourselves.
A
You
can
just
log
to
the
console,
so
one
of
the
teams,
both
their
logging
sidecar
and
basically
you
deploy
this
as
a
daemon
set
and
all
your
services
can
just
log
to
a
port
on
their
local
host
and
the
logs
and
the
matrix
organ
streamed
off
to
some
stream
like
pops
up
or
Kafka.
But
of
course
we
benefited
from
many
other
systems
and
components
that
we
bought
by
other
teams
and
within
Gajic.
A
One
of
the
things
I
wasn't
showing
earlier
was
that
there
are
these
databases
actually
with
deployed
with
the
models
and
these
databases
or,
for
example,
radius
paws
that
have
data
loaded
into
them
through
the
jobs
that
are
executed
within
the
cuber
Nettie's
environment,
so
they'll
take,
for
example,
a
data
set
called
GCS
and
load
it
into
a
raid
ISM
and
just
finish
their
execution.
And
then
this
Redis
contains
all
the
features
for
that
specific
model.
To
actually
do
it
inference
because
the
request
needs
to
be
enriched
for
those
feast
features.
A
So
the
tracks,
where
a
lot
of
the
panes
that
we
had
there,
it
can
intelligently
schedule,
for
example,
a
workload
that
means
a
GPU
and
it
finds
the
right
node
with
the
right
amount
of
memory
to
run
your
compute,
so
among
cube
really
benefits
from
its
capabilities
in
terms
of
workloads.
But
one
of
the
inefficient
parts
here
was
the
duplication
of
features
within
these
latest
pods.
So
we
were
having
a
lot
of
issues
and
a
lot
of
pain
around
feature
engineering
data
scientists
would
reinvent
the
same
features
across
projects
across
teams.
A
A
So
we
knew
there's
a
hot
spot
that
we
wanted
to
fix,
and
so
we
looked
at
building
a
feature
store
for
this,
so
there
are
two
to
basically
roles
or
personas
that
use
a
feature
store,
but
one
one
side:
you
have
the
data
creators
they're,
creating
datasets
from
a
raw
data,
electronic
transformations
and
batch
and
they're
creating
extreme
transformations
when
I
publishing
it
in
some
stream.
And
then
you
have
a
user
of
that
data
that
wants
to
train
a
model
and
somehow
serve
that
model.
A
So
what
we
ended
up
building
was
a
store
that
allows
you
to
ingest
these
features
either
real-time
or
batch,
and
they
get
stored
in
from
these
streams
into
two
stores.
One
is
a
feature
warehouse
in
our
case.
That's
bigquery,
the
other
is
a
serving
store
which
can
be
either
BigTable
or
register
pending
on
your
requirements
and
what
this
provides
to
the
end
user.
They
Mel
engineer
is
a
consistent
API
to
both
of
these
stores,
so
they
define
a
feature
set
and
then
they
can
query
from
the
future.
A
Warehouse
over
a
time
range,
historical
feature,
data
train,
their
model,
take
their
model
into
production
and
then
use
the
same
feature
say
to
query
real
time
features
in
serving,
and
so
this
eliminates
the
serving
training
skew.
That
means
you
can
manage
your
features
centrally,
so
you
don't
have
duplication
between
teams
and
they
can
discover
features
that
are
in
use,
and
it
also
means
you
don't
need
to
roll
out
new
databases
for
each
model
that
gets
deployed
so
one
of
the.
A
So
this
is
all
the
architecture
changes
when
you
have
a
feast
serving
deployment,
so
you'll
just
deploy
that
piece
serving
within
the
same
cluster
with
which
is
managing
your
production
workload.
And,
of
course
you
could
have
deployed
that
anywhere.
That
could
be
in
a
managed
service
or
it
could
be
on
a
VM.
But
ultimately,
one
of
the
benefits
to
Kuban,
alias
and
running
in
all
systems
is
that
you
can
deploy
everything
basically
in
one
cluster.
So
if
you
have
a
page
of
going
over
to
a.m.
A
you
can
just
look
at
your
one
cluster
to
get
a
complete
overview
of
all
the
issues
that
have
there
and
debug
that
so
from
an
operational
standpoint,
is
a
big
benefit
to
standardizing
on
kubernetes
for
for
these
email
systems,
and
then
we
had
a
rapid
expansion.
So
we
were
deploying
more
and
more
models
and
systems
to
these
model
variants
within
a
specific
service
type,
so
go
ride,
is
our
ride-hailing
service
type,
but
we
had
many
different
service
types
that
we
were
expanding
to.
So
we
had
free
delivery,
which
also
needs
allocation.
A
We
have
logistics,
we
have
ride-hailing
on
cars
and
they
will
have
their
own
different
models
because
it's
trained
on
different
data,
so
we
were
expanding
rapidly
into
all
these
service
types
and
then
we're
also
expanding
into
new
markets,
so
we're
expanding
into
Singapore
and
Thailand
and
Vietnam
as
well
so
quickly
the
amount
of
models
and
systems
grew
exponentially
and
you
can't
just
duplicate
these
models
and
just
serve
from
a
single
endpoint.
For
example,
in
Thailand
there
are
very
strict
regulations
on
what
taxi
motorcycle
taxis
are
allowed
to.
A
A
A
A
So
whenever
one
of
these
artifacts
changes,
we
are
obtaining
the
deployed
system,
and
this
is
a
very
scalable
approach
across
all
of
our
markets,
and
the
biggest
benefit
of
this
is
that,
ultimately,
you
can
tie
your
experimental
outcome
to
the
unique
combination
of
artifacts
that
went
into
that
specific
deployment.
So
it
allows
you
to
reason
about
why
a
deployment
worked
or
didn't
work
and
then
to
add
to
this.
What
we're
focusing
on
now
and
what
the
cupula
team
is
looking
at
now
is
metadata
management.
A
How
can
you
go
hockey
track
all
the
metadata
across
this
lifecycle
and
then
go
up
stream
when
you
discover
something
went
well
or
didn't
go
well,
and
then
it
gives
you
more
context
about
that,
so
just
to
cover
the
good
parts
that
we
discussed
earlier.
The
the
ecosystem
currently
for
ml
and
kubernetes
is
vibrant
and
that's
one
of
the
biggest
draws
for
us,
especially
with
cube
flow,
bringing
together
all
these
these
packages
and
components
and
frameworks,
the
consistent
API
and
the
q4q
Bernese
provides
is
a
great
way
to
extend
and
both
the
platform's.
A
A
But
we
want
to
address
the
long
tail
of
customers
that
are
throughout
our
organization,
because
we
have
thousands
of
employees
and
we're
also
focusing
on
metadata
tracking,
like
I
said
earlier,
and
also
integrating
sto
tracing
into
our
stack
and
I'll.
Leave
you
on
this
quote
from
Kelsey
and
ITAR,
which
I
think
is
very
cool,
at
least
in
our
experience.
Building
a
name
of
that
for
holding
an
email
platform
on
top
of
cube,
really
made
sense.
B
A
B
You
very
much
well
that's
a
really
great
introduction
and
motivation
for
coop
flow,
because
what
we
realize
talking
to
a
lot
of
companies
and
customers
is
that
gojek
wasn't
alone
in
building
a
platform
for
machine
learning
and
using
kubernetes
lots
of
people
were
doing
that
same
thing.
You
know,
and
they,
as
william
mentioned.
B
You
know
they
built
a
custom
resource
lasso
internally
for
doing
graphs
of
inference
of
models
when
there
were
companies
outside
that,
like
Seldon
doing
the
exact
same
things
and
many
of
those
companies
have
presented
here
at
coop
kong,
and
so
what
we
decided
to
to
do
is
we
said
why
not
create
an
open
platform
for
machine
learning
on
kubernetes
that
everyone
can
contribute
to
and
everyone
could
use
and
that's
what
coop
flow?
Is
it's
an
open,
kubernetes
native
platform
for
our
mouths?
B
Your
features
there
and
that's
going
to
allow
your
data
scientists
to
rapidly
develop
and
iterate
on
models
and
we're
going
to
show
you
how
they
can
do
that
using
notebooks
running
on
coop
flow
use
on
top
of
kubernetes,
and
so
after
we've
developed
our
model.
We're
gonna
take
advantage
of
fairing,
which
is
a
library
in
kubernetes.
B
So
this
is
the
central
dashboard
for
our
coop
flow,
and
so
the
idea
is
that
we're
going
to
deploy
and
manage
a
bunch
of
applications
for
machine
learning
on
kubernetes
right
and
so
after
you've
logged
into
coop
flow.
You
can
use
this
dashboard
to
navigate
between
all
the
different
applications
and
right
now
we
basically
have
three
applications.
We
have
notebooks,
sorry,
that's
the
docs.
This
is
notebooks.
We
have
Captain,
which
is
our
hyper
parameter
tuning
system,
and
then
we
have
pipelines.
B
We
also
have
a
bunch
of
components
like
custom
resources
for
training
and
deploying
models,
but
those
don't
necessarily
have
you
eyes
so
I'm
not
showing
them
right
now.
So
let's
go
to
notebooks
right
so
from
here
we
can
see
all
of
our
notebooks
that
we
have
running,
and
so
we
make
it
really
easy
for
data
scientists
to
launch
and
run
manage
multiple
notebooks
simultaneously.
B
So
they
could
have
different
environments,
maybe
one
for
pi
torch,
maybe
one
for
tensor
flow
and
maybe
ones
running
some
long-running
job
and
they
can
work
another
one
and
so
to
launch
a
notebook.
They
just
click
here
on
new
server
and
they
have
a
simple
web
UI
where
they
can
fill
out
a
form
in
order
to
launch
a
notebook.
So
they
just
give
it
a
name
and
then
they
can
specify
the
docker
image
they
want
to
use.
B
Storage
systems
like
an
NFS
share
so
after
you've
done
that
you
can
go
ahead
and
connect
to
your
notebook,
so
I'm
going
to
switch
over
to
the
notebook,
I've
already
started
up,
and
so
this
is
a
notebook
that
we're
going
to
use
for
our
demo
and
so
in
this
demo.
What
we're
going
to
be
doing
is
we're
going
to
be
using
the
Chicago
taxicab
data
set,
which
is
a
data
set,
that's
used
in
a
lot
of
ml
research
and
examples,
and
it's
basically,
a
data
set
of
Chicago
taxi
rides.
B
It's
got
various
features
about
those
taxi
rides
such
as
duration,
pickup,
time
delivery
destination
and
then
fare
cost
how
much
the
taxi
ride
cost,
and
so
what
we're
going
to
try
to
do
is
we're
going
to
try
to
use
these
features
to
predict
how
much
a
taxi
ride
is
going
to
cost.
Let
me
zoom
in
how's
that
font
size.
Can
people
see
that
okay,
we're
good,
alright,
so
I'm
going
to
scroll
down
here
or
is
it
so
so
here?
B
What
we're
doing
is
we're
going
to
load
some
data
set
load,
some
data
from
a
CSV
file
that
contains
the
Chicago
taxicab
data
set,
and
so,
in
this
case
we're
just
loading
it
into
a
panda's
data
frame.
So
we're
going
to
load
this
data
into
a
feast,
that's
mostly
to
illustrate
how
you
can
actually
go
about
loading
data
into
feast
in
practice.
You'd
be
continually
streaming
or
ingesting
your
data
from
like
your
logs
into
your
data
store
or
fees,
but
we
wanted
to
sort
of
illustrate
this
aspect
of
feast
for
you.
B
So
after
we've
loaded
that
data,
what
we're
doing
here
is
we're
basically
computing
some
derived
features
where
we're
basically
using
pandas
and
and
data
frames
to
combine
several
features
and
compute
some
other
features,
but
and
that
we're
going
to
load
into
the
data
store
and
so
down
here.
This
is
where
we
start
to
define
an
importer
to
actually
import
the
data
into
feast
and
I'm
going
to
turn
it
back
over
to
warm
to
basically
describe
what's
happening
here
so.
A
What
the
importer
is
doing
is
it's
taking
a
data
frame
as
an
input,
and
then
it
allows
you
to
define
some
information
about
the
entity
which
in
this
case
is
the
right
and
it
allows
you
to
define
the
mapping
of
the
columns
within
that
data
frame.
Two
features
that
exist
within
the
feature
store
and
if
I
don't
exist,
then
you
can
specify,
you
can
see,
it
says,
apply
features
it'll
actually
create
those
features.
The
first
time
you
run
this
import.
A
So
when
your
run
list,
it's
going
to
start
a
job
in
the
background
and
it's
going
to
load
in
all
of
those
features
and
like
Jeremy
said
in
production,
this
would
probably
be
at
the
end
of
an
ETL
or
you'd
run
a
similar
job,
that's
long-running
to
streaming
data
constantly
from
a
stream,
and
so
your
feature
stores
always
being
updated
with
new
data.
And
if
you
go
down
a
little
bit,
you'll
see.
Ok,
it's
creating
all
of
these
features
within
within
the
future.
A
Store
I'll
just
want
to
scroll
down
a
little
bit
right
and
then
you
get
to
the
most
important
part
which
is
defining
your
feature
set,
and
this
is
what
is
going
to
allow
us
to
create.
You
could
provide
us
with
a
way
to
and
get
consistency
between
our
training
and
serving
so
in
this
case,
we're
defining
a
feature
set,
which
is
a
list
of
features
by
their
ID
and
then
we're
going
to
create
an
object
out
of
that.
A
That's
part
of
the
tasty
Kay
and
then
we're
going
to
materialize,
a
dataset,
so
we're
going
to
retrieve
the
features
from
the
training
store
and
if
you
you'll
see
that
we
did
ahead
there
and
it's
the
same
data
that
we
just
ingested
and-
and
this
is
this-
is
what
the
feature
store
provides
for
you.
It
allows
you
to
query
over
a
time
range
and
pull
out
and
materialize
that
data
for
training,
and
importantly
it
unit
you're,
going
to
use
that
feature
set
again.
That
lists
the
features
for
on
the
serving
side.
A
B
B
You
know
a
bunch
of
features
from
you
know
their
data
store
like
feast
and
then
they
would
try
to
visualize
that
data
in
order
to
see
you
know,
is
there
a
signal
there
that
I
can
use
to
solve
this
problem
and
so
to
do
that
we're
going
to
use
tensorflow
data
validation.
So
this
is
one
of
the
many
applications
that's
in
tf-x,
which
is
a
set
of
applications
that
Google
is
open
sourcing
and
releasing
to
enable
you
to
build
complex
ml
systems,
and
so
what
TF
DB
does
is.
B
It
will
go
through
your
data
set
and
compute
various
statistics,
and
then
it
provides
a
nice
handy,
widget
I
think
it's
based
on
facets
that
allows
you
to
visualize
those
statistics
right.
So
here
we're
looking
at
you
know
various
columns,
like
passenger
count
and
it's
printing
out
some
various
statistics,
and
then
we
can
visualize
things
like
the
distribution
and
it's
interactive.
B
So
after
we've
done
that,
then
we
can
go
down
here
and
we
can
start
to
define
the
code
to
train
our
model
and
so
here's
the
training
code,
and
so
you
can
see
down
here.
We
actually
there
we
go.
We
load
the
data
from
feast
and
here
what
we're
doing
is
we're
specifying
the
time
range
that
we
want
to
download.
B
Here
we
have
the
predict
code
and
what
the
predict
code
is
doing
is
it's
taking
in
the
eye,
that's
identifying
that
the
taxi
ride
and
then
from
feast
we're
gonna
load
that
features,
so
you
can
see
we're
just
calling
get
serving
data,
and
here
we're
gonna
load
in
the
same
features
that
we
were
using
and
training
and
then
we're
gonna
feed
that
data
into
our
linear
model.
So
the
most
important
thing
is
that
what
you
don't
see
here,
so
what
you
don't
see
here
is
a
lot
of
complicated
code
to
go
and
fetch
these
features.
B
These
different
features
from
multiple
different
databases
and
compute
the
features
from
the
raw
data,
and
so
as
well
I
mentioned
before
that
allows
us
to
avoid
the
training
serving
askew
because
we're
getting
using
the
same
features
in
training
as
in
serving
and
that's
because
of
we're
using
fees
to
take
care
of
that.
So
after
we've
done
that,
we
can
go
ahead
and
train
our
model
inside
our
notebook.
So
here
we're
just
in
invoking
train.
B
So
you
can
imagine
if
you're
training
on
a
very
small
data
set
so
that
you
can
train
quickly
and
iterate
once
you've
got
a
model
that
you're
happy
with
you
might
want
to
go
ahead
and
launch
that
as
a
long-running
job
on
kubernetes.
Maybe-
and
maybe
you
know,
process
the
full
data
set,
so
maybe
you've
been
training
only
in
the
last
28
days,
and
now
you
want
to
go
and
train
on
all
of
your
data,
maybe
for
the
last
year.
So
in
this
case
we
can
use
fairing
to
easily
launch
a
training
job
on
kubernetes.
B
So
to
do
that,
all
we
have
to
do
is
specify
to
two
values.
One
is
the
docker
image
that
we
want
to
use,
and
so,
in
this
case
we're
going
to
use
the
same
runtime
environment
as
we're
using
our
notebook
for
our
notebook.
So
we
can
get
the
same
environment
as
we're
using
in
our
notebook
and
then
the
other
thing
we
have
to
specify
is
the
docker
image
where
we
want
to
push
the
dock,
push
the
image
and
then
with
Baron.
B
B
We
can
then
just
invoke
a
couple
lines
of
Python
code
to
actually
launch
a
kubernetes
job
and
then
baring
will
actually
go
and
fetch
the
logs
from
that
kubernetes
job
and
print
them
out
for
convenience,
and
we
can
actually
call
out
to
kubernetes
to
print
out
the
actual
kubernetes
job
that
we
actually
submitted
and
ran.
So
we
can
see,
we
just
have
the
expected
spec
for
Cooper
net
equipment,
his
job
once
we've
trained
our
model
me.
B
What
me
might
want
to
actually
go
ahead
and
deploy
that
in
kubernetes
to
get
a
restful
endpoint
that
we
can
call.
So
in
this
case
we
can
again
use
fairing
to
automatically
launch
a
build
and
launch
deployment
on
kubernetes,
and
that's
just
another
couple
lines
of
Python
code
to
deploy
that
and
then
you
can
see.
B
We
printed
out
the
endpoint
that
we're
getting
inside
our
cluster
to
actually
do
predictions
and
then,
finally,
as
as
William
showed,
we
might
want
to
develop
a
complex
pipeline
to
run
multiple
steps
and
to
make
it
easy
to
repeat
all
these
steps.
So
for
that
we
can
use
coop
flow
pipelines
and
so
to
define
a
pipeline.
You
know
we
just
define
a
function
which
we're
going
to
use
to
define
the
steps
in
our
pipeline,
and
we
add
this
simple
annotation
to
let
coop
flow
pipelines
know
this
is
actually
a
pipeline.
B
Then
inside
the
pipeline
we
just
define
the
steps
of
our
pipeline.
Each
step
in
a
pipeline
is
going
to
run
a
containerized
operation.
So
basically,
it's
going
to
spin
up
a
pod,
and
so
all
we
have
to
do
is
specify
the
image
we
want
to
use,
and
so,
in
this
case
we're
going
to
use
the
same
images
we
used
for
training
and
deploying
and
that
were
created
by
fairing.
And
then
we
just
specify
the
the
command
that
we
want
to
invoke.
And
so
in
this
case
we
can
just
tell
it
that
we
wanted
run
training.
B
And
then
we
can
tell
what
time
range
of
data
that
we
actually
want
to
train
on.
And
we
can
pass
that
as
command-line
arguments.
Sorry
and
then
to
actually
submit
it.
The
pipeline.
We
can
just
call
out
to
the
pipe
of
the
coop
flow
pipelines,
SDK
and
that
will
go
ahead
and
submit
the
pipeline
and
print
out
a
link,
and
we
can
open
that
link
to
see
that
pipeline
running
in
coop
flow
pipelines.
And
so
in
this
case
we
have
a
very
simple
pipeline
where
we're
just
pre-processing
and
training
and
then
validating
the
data.
B
B
Just
a
couple
quick
words
about
coop
flow,
so
this
is
kind
of
the
motivation
for
coop
flow.
So
if
you're
building
coop
flop
ml
pipelines,
it's
a
complex
workflow
that
you
have
multiple
steps
that
you
go
through
to
build
and
deploy
ml
into
production.
So
you
have
data
ingestion.
You
have
training,
you
have
deploying
for
each
of
these
steps
in
your
process.
You
have
multiple
applications
that
you
could
choose
from.
So,
for
example,
for
serving
you
could
be
using
tensorflow
serving.
B
You
could
be
using
invidious
inference
server
so
with
coop
coop
flow,
we're
trying
to
make
it
very
easy
to
run
these
applications
on
kubernetes
and
then
combine
them
into
complex
workflows.
We
have
three
core
tenants.
You
know
one
is
composability,
so,
as
you
saw
from
our
our
demo,
we
want
to
make
it
super
easy
to
take
different
applications
like
feast
and
tf-x
and
combine
them
into
complex
workflows,
we're
also
very
interested
in
scalability.
So
we
want
to
make
it
super
easy
for
users
to
scale
out.
B
We
want
to
make
it
super
easy
to
run
anywhere
that
kubernetes
runs
and
that's
one
of
the
and
that's
one
of
the
advantages
of
using
kubernetes
is
that
it
runs
in
the
cloud
and
on-premise
is
a
quick
diagram
of
what
coop
flow
looks
like
the
architecture.
We
basically
have
a
bunch
of
shared
applications
and
custom
controllers
that
run
in
a
shared
namespace.
B
But
then
all
users
can
deploy
and
consume
coop
flow
in
their
own
namespace,
so
they
can
run
pipelines
and
notebooks
in
their
own
namespace,
and
then
we
expose
all
these
services
behind
an
ingress
so
that
you
can
access
them
from
out
the
cluster
as
I
mentioned
before
we
are
an
open-source
project
and
we
have
a
weird.
We
have
a
lot
of
momentum
and
a
lot
of
great
contributions
from
outside
the
community
and
from
within
the
community.
So
we
have
over
30
companies
participating
and
we're
growing
rapidly,
as
these
graphs
show.
B
B
Is
we
want
to
have
a
low
bar
so
make
it
super
easy
for
people
to
get
started
with
machine
learning
on
kubernetes,
but
then
we
want
to
have
a
very
high
ceiling,
so
we
don't
want
to
limit
what
people
can
do
with
machine
learning
on
kubernetes
and
so
finally,
I
think
it
would
be
great
if
you
went
to
the
coop
flow
docs
page
and
found
our
getting
started
guide
to
try
out
coop
flow.
You
can
also
go
to
the
feast
website
and
follow
the
instructions
to
installing
feast.
B
And
if
you
go
to
this
github
page,
you
can
actually
download
the
demo
that
we
showed
you
today
and
go
through
it
yourself.
There's
a
lot
of
great
talks
to
here
at
coop
con
about
coop
flow.
You
might
want
to
consider
checking
out
some
of
the
videos
or
materials
online.
Thank
you
very
much.
I.
Don't
think
we
have
any
time
left
for
questions,
but
maybe
we
can
try
for
one.
If
anybody
has
a
quick
one.