►
From YouTube: Building and Managing a Centralized Kubeflow Platform at Spotify - Keshi Dai & Ryan Clough, Spotify
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Building and Managing a Centralized Kubeflow Platform at Spotify - Keshi Dai & Ryan Clough, Spotify
Machine learning workflows within Spotify have been migrated to Kubernetes by adopting Kubeflow and Kubeflow Pipelines. It helps teams increase model development speed and reduce the time to productionize a machine learning model. In this talk, we will demonstrate some best practices Spotify has learned from managing Kubernetes for backend services and apply them to building a centralized Kubeflow platform. We treat infrastructure as code. We establish customizable and repeatable deployment process. Even with a handful of machine learning/data engineers, we are successfully able to manage multiple Kubernetes clusters and machine learning workloads at scale. We will also show how teams at Spotify use Kubeflow platform as a one-stop shop for their machine learning development, which helps them build better products to improve user listening experience.
https://sched.co/UaWi
A
All
right,
hi
everyone
thank
you
for
coming.
So
this
is
our
talk.
Coop
float,
Spotify,
building
and
managing
a
centralized
platform.
It's
not
gonna
be
as
technical
of
a
talk.
It's
a
little
bit
more
of
a
case
study
of
how
we
deploy
coop
flow
at
our
company
and
the
sort
of
trials
and
tribulations
and
things
we
have
to
consider
along
the
way.
I
hope
that
you
can
take
away
something
if
you're
not
using
coop
flow,
yet
yourself
that
you
can
maybe
take
something
away
about
how
you
might
use
it
in
your
own
organization.
A
If
you
haven't
heard
of
us
before
we
are
a
music
streaming
service,
we
also
have
podcasts
and
we
launched
in
2008.
We
have
248
million
active
users,
50
million
tracks
on
our
platform
and
70
were
launched
in
79
countries.
We
use
machine
learning
all
over
our
products,
most
notably
in
our
recommendations.
A
Products
such
as
discover
weekly,
which
is
a
weekly
personalised
playlist,
that
we
provide
for
our
users,
but
it's
also
in
many
other
places,
such
as
our
search
functionality,
our
homepage
and
much
more
I'm,
making
the
assumption
that
a
lot
of
you
know
what
coop
flow
is,
but
in
case
you
don't.
The
brief
elevator
pitch
is
that
coop
flow
is
an
open
source
framework
for
running
machine
learning,
pipelines
on
kubernetes.
It
transforms
individual
components
of
the
ml
workflow,
such
as
training,
transformation,
model,
evaluation,
etc
into
individual
docker
containers
that
can
be
chained
together.
A
So
let's
talk
a
little
bit
about
the
organizational
structure
at
Spotify.
I
know
this
is
a
kubernetes
conference
so
wide
talk
about
this
if
you're
familiar
with
Conway's
law,
it's
that
software
systems
tend
to
reflect
the
structure
of
the
organization
in
which
they're
built
and
so
I
don't
want
you
to
go
back
to
your
office
on
Monday
and
say:
let's
adopt
coop
flow
and
let's
deploy
it
this
way,
because
this
is
the
way
that
Spotify
did
it
and
it
works
for
them.
A
I
want
you
to
see
the
trade-offs
that
we
made
and
how
they
relate
to
our
organizational
structure
and
use
that
to
inform
your
own
decision
making
process
as
to
whether
this
would
be
right
for
you
and
what
kind
of
trade-offs
that
you
need
to
make
so
we're
gonna
start
the
story
with
you.
I'm
gonna
put
you
in
our
shoes
you're
now
an
engineer
at
Spotify.
Congratulations
on
your
new
job,
you're
on
you're
on
a
team
and
teams
at
Spotify.
A
We
call
squads
and
your
squad
is
machine
learning
infrastructure,
the
team
that
Keshi
and
I
work
on
you
are
one
squad
of
280
squads
all
across
the
globe.
We
have
offices
in
Stockholm
New,
York
City,
which
is
where
Keshi
and
I
are
London,
Boston
and
more,
and
these
squads
sort
of
bubble
up
in
a
short
hierarchy
into
business
units,
for
example
our
free
products,
our
premium
product
and
other
business
units.
A
Some
smaller
percentage
of
these
squads
use
machine
learning
and
they're.
Now
your
customers
as
a
machine
learning
infrastructure
engineer.
So
what
does
a
typical
customer
team?
Look
like
squads
at
Spotify
are
small
cross-functional
and
what
we
call
responsibly
autonomous.
So
a
given
team
may
be
constructed
of
any
number
of
back-end
Engineers
front-end
data
scientists,
data
engineers,
ml
engineers,
etc,
based
on
the
particular
product
or
feature
that
they're
working
on.
A
This
autonomy
and
decentralization
is
a
trade
off
for
Speed
versus
coordination.
We
trust
teams
to
make
the
decisions
that
make
the
most
sense
for
them
and
in
order
to
execute
as
fast
as
possible,
and
this
means
that
teams
are
not
blocked
on
needing
some
resource
from
another
team.
They
should
be
staffed
in
accordance
to
what
they
need
to
deliver
their
feature.
A
The
cons
of
this
is
that
there's
a
lack
of
standardization.
If
you
work
in
the
m/l
space,
you
would
know
that
there's
a
lot
of
different
tools
that
you
can
use
in
ml
and
new
ones
are
coming
out.
All
the
time
and
autonomous
teams
means
that
all
the
different
machine
learning
teams
historically
will
be
using
a
different
tool
set,
and
this
leads
to
a
sort
of
information
or
experience
siloing.
A
So
there
are
maybe
teams
that
have
been
around
a
while
and
built
infrastructure
around
the
tools
that
they
use
and
younger
teams
that
are
just
starting
to
use.
Machine
learning
may
choose
to
use
different
tools
that
are
technologically
incompatible
with
the
infrastructure
that
they
built
for
themselves
and
then
there's
also
the
problem
of
you
know.
We
have
teams
in
different
time
zones
in
different
countries
and
different
business
verticals
that
may
not
have
exposure
to
what
other
teams
are
doing.
A
So,
given
this
organizational
structure,
you're
posed
with
a
question
for
you
as
a
machine
learning
infrastructure
team,
is
it
better
for
you
to
create
a
centralized
resource
for
your
customers
to
use
and
take
on
that
operational
responsibility?
Or
is
it
better
for
you
to
make
the
self
deployment
use
case
as
easy?
As
possible
for
your
customer
teams,
we're
gonna
start
by
considering
the
self
deployment
scenario,
because
for
an
autonomous
team,
this
seems
like
the
natural
choice.
It's
very
common.
It's
modify
for
teams
to
own
and
operate
their
own
resources.
A
We,
as
I
mentioned
before.
We
trust
teams
to
know
what's
best
for
them,
so
they
know
what
sort
of
scale
and
cost
that
they
need
to
do
to
to
have,
and
they
don't
have
to
worry
about
fighting
over
resources
within
the
cluster
or
worrying.
If
some
other
team
is
gonna,
do
an
upgrade
that
will
break
their
production
pipelines.
So
what
does
your
job
look
like
in
this
kind
of
a
scenario?
A
Well,
first
and
foremost,
you're
going
to
want
to
deploy
coop
flow
yourself
and
figure
I
figure
out
what
kind
of
customizations
are
needed
to
fit
within
your
infrastructure
or
networking
system
etc,
and
then
use
what
you've
learned
to
provide
a
tool
or
a
script
to
make
deployment
easy?
Then
you
want
to
build
on
top
of
coop
flow
and
provide
common
components
for
em
tasks
that
your
teams
are
going
to
want
to
use
or
components
that
will
integrate
with
maybe
internal
infrastructure
that
you
already
have
that
maybe
isn't
publicly
available.
A
This
is
particularly
attractive
because
it's
the
least
amount
of
work
for
you
and,
as
we
know,
good
engineers
do
just
as
much
work
as
they
need
to.
In
this
case,
you
don't
have
operational
responsibilities:
you're
not
caught
on
call
at
2:00
a.m.
when
someone's
pipeline
breaks,
there's
no
long-term
resource
planning
around.
You
know
how
do
I
make
sure
that
my
cluster
is
gonna
scale
to
the
future
of
machine
learning
at
Spotify,
etc.
A
And
while
you
can
minimize
that
by
providing
say
a
deployment
tool
or
and
the
fact
that
coop
flow
adds
Drax
away
a
lot
of
the
kubernetes
details,
they're
still
going
to
be
the
case
where
all
of
these
individual
teams
have
production
pipelines
running
and
something
at
some
point
is
gonna,
go
wrong
on
their
clusters
and
they're
gonna
need
to
debug
it,
and
if
you
made
things
too
easy
for
them,
they're
not
gonna,
know
enough
about
Cooper
nineties
to
debug
it.
They
might
be
calling
you
anyways
for
help.
A
The
next
situation
is
you
need
to
consider
how
this
scenario
is
going
to
work
out
long
term.
So
we
already
have
this
information
siloing
problem,
where
individual
teams
build
up
expertise
and
infrastructure
and
if
they
have
their
own
coop
flow
clusters,
you're
still
going
to
be
hiding
the
work
of
teams
that
may
not
have
a
natural
level
of
exposure
to
other
teams.
We
have
some
inspiration
within
Spotify
already
for
this
sort
of
problem.
A
We
have
a
centralized
developer
platform
that
you
can
log
on
to
and
see
information
about
all
sorts
of
pipelines
and
back-end
services
that
Spotify
you
can
see
their
status.
So
you
can
see
what
team
owns
it.
You
can
see
where
the
code
is
for
that
service
or
pipeline,
etc.
I
think
you
would
be
really
lucky
as
an
infrastructure
engineer,
to
expose
the
same
kind
of
information
about
machine
learning
within
your
organization.
So,
for
example,
a
team,
that's
working
on
a
new
model
can
look
at
the
work
of
other
teams.
A
They
can
look
at
what
datasets
people
are
using,
how
people
are
transform
the
features
from
that
dataset
etc,
and
that's
gonna
be
really
hard
to
do
if
every
team
has
their
own
cluster
with
their
own
metadata
store,
and
you
have
to
figure
out
how
to
coalesce
all
of
that
information
and
get
the
permission
from
everybody
else's
resources.
Mm-Hmm.
A
A
But
the
situation
here
is
that
you'll
often
have
breaking
changes
and
in
an
autonomous
team
scenario,
you
can't
force
a
team
to
upgrade,
and
so,
if
your
team
is
trying
to
build
on
top
of
coop
flow
and
make
assumptions
about
what
kind
of
features
are
there,
what
version
people
are
running?
You
need
to
worry
about
whether
or
not
you
can
get
teams
to
upgrade
to
use
your
new
features
and.
A
A
A
The
story
goes
like
this
at
Spotify
we're
entirely
on
GCP,
but
this
concept
exists
in
many
other
cloud
providers,
because
we're
autonomous
every
team
has
their
own
individual
GCP
project
and,
as
you
might
imagine,
it's
common
that
services
that
one
team
build
is
going
to
want
to
communicate
to
services
other
internal
services
at
the
company.
The
problem
is
that
by
default,
because
each
of
the
products
have
their
own
networking
isolation,
you
can't
do
this
so
to
solve
this
problem,
we
use
what's
called
a
shared
V,
PC
or
virtual
private
cloud.
A
It's
a
shared
networking
layer
that
individual
services
can
connect
to
to
communicate
with
each
other.
So
you
might
be
thinking.
Okay,
you've
solved
this
problem,
you
have
the
share
of
e
PC.
Just
have
the
teams
add
their
kubernetes
clusters
to
it
and
that's
the
end
of
the
day.
The
issue
is
that
this
is
actually
a
pretty
high
friction
process,
so
every
individual
team
now
in
the
future
and
if
we
have
restructuring,
is
going
to
have
to
talk
to
security
because
you're
attaching
to
a
networking
layer
that
exposes
internal
services.
A
To
ask
every
individual
team
to
do
this
themselves
is
really
high
friction
and
if
you're,
an
infrastructure,
team
and
teams
don't
have
to
use
the
tools
that
you
make.
This
is
a
huge
blocker
for
teams
that
might
just
want
to
keep
using
what
they
already
have.
You
want
to
try
and
optimize
a
bit
for
adoption
of
your
tools.
So,
given
that,
let's
look
at
the
pros
and
the
cons
of
the
centralization
use
case
number
one,
our
shared
V
PC
problem
is
solved
once
people
don't
have
to
worry
about
it.
A
Number
two
all
of
the
kubernetes
expertise
is
now
centralised
on
our
team
T.
That
means
that
teams
no
longer
have
to
worry
as
much
about
the
infrastructure
underlying
koop
flow
and
that's
the
whole
point
of
coop
flow
teams
can
focus
on
machine
learning
and
not
the
infrastructure.
Lastly,
or
not.
Lastly,
that
means
you
get
easier
adaptation
from
the
teams.
A
I'd
be
remiss
not
to
mention
the
downsides
of
this
situation,
which
is
number
one
we're
a
pretty
small
team,
and
now
we
have
this
big
operational
responsibility.
We
have
to
go
on
call
for
it.
We
have
to
make
LoZ,
but
this
con
may
not
be
so
much
of
a
con
when
you
consider
the
fact
that
it's
either
us
taking
the
burden
of
this
responsibility
or
every
single
individual
team,
which
in
that
case,
you
might
not
consider
it
a
con.
Someone
has
to
take
that
burden
and
has
an
infrastructure
team.
A
A
If
one
team
is
going
to
accidentally
hog
up
the
resources
from
another
team
and
then
the
upgrade
scenario,
if
you
upgrade
kubernetes
or
you
upgrade
coop
flow,
if
you
break
people's
pipelines,
you're
breaking
all
of
machine
learning
at
Spotify,
not
just
one
team
and
then
lastly,
coop
flow
is
not
yet
fully
multi-tenant,
but
that's
coming
soon,
and
we
really
look
forward
to
that-
hopefully
early
next
year,
so
given
some
centralized
resources.
So,
as
I
mentioned
before,
we
have
decided
to
centralize,
given
these
pros
and
cons.
So
given
that,
how
can
we
optimize
the
developer?
A
Experience
number
one
as
I
alluded
to
before
we
provide
a
number
of
common
components.
We
standardize
on
tf-x
or
tensor
flow,
extended.
It's
a
framework,
that's
opinionated
and
provides
a
number
of
pre-made
components
and
runs
on
coop
flow.
We
have
our
own
internal
fork
that
utilizes
all
of
the
stuff
that
they've
written
for
common
tasks.
A
So
we
provide
our
users
with
a
template
repo
that
they
can
clone
and
we'll
fill
in
a
lot
of
the
pre-existing
information
about
the
centralized
cluster,
and
this
reduces
a
lot
of
boilerplate
helping
adoption
and
then
we
also
provide
a
thin
CLI
tool
that
utilizes
once
again
a
lot
of
the
assumptions
around
the
centralized
resources.
So
we
know
how
authentication
should
work.
We
know
what
clusters
things
should
be
submit
to.
A
We
can
also
enrich
the
metadata,
that's
being
when
they
submit
a
coupe
flow
pipeline
to
our
infrastructure
and
then,
lastly,
as
an
added
benefit,
we
can
build
other
tooling
into
this,
such
as
scaffold
our
users,
don't
have
to
write
or
compile
or
build
docker
containers
themselves.
They
merely
interface
with
the
Python
DSL
and
scaffold
will
build
the
docker
containers
for
them
as
part
of
pipeline
compilation.
B
B
Our
story
of
building
a
centralized
cube
flow
platforms
that
made
this
year
and
it
has
gone
through
several
iterations.
In
the
very
beginning
we
started
with
out
of
box
default
deployment,
or
later
we
switched
to
terraform
to
create,
create
and
manager,
TCP
resources
and
used
caisson,
yet
to
deploy
key
flow
apps,
eventually,
with
cube
flow
0.6,
we
replaced
a
SONET
who
is
customized
next
I
wish
you
some
happy
experience
as
well
as
some
pin
points
in
this
journey.
B
B
The
default
deployment
tool
comes
with
a
nice
web
UI
with
a
few
parameters.
It
builds
a
cube
flow
cluster
with
a
public
URL
through
Google
Cloud
endpoints,
so
people
can
access
it
immediately
after
its
created.
Meanwhile,
the
says
is
protected
by
IEP
Google
identity
aware
proxy
service,
so
it
will
be
as
secure
as
our
Google
services.
B
So
this
tool
is
really
really
simple.
It
seems
perfect
for
us,
since
we
run
everything
onto
UCP.
However,
it
might
be
too
good
to
be
true.
First,
the
whole
process
is
a
black
box.
We
didn't
really
know
what's
going
on
behind
the
scene,
so
it
was
impossible
for
us
to
do
any
customization.
All
we
could
do
was
waiting
for
the
cluster
to
magically
show
up,
and
second,
it
has
no
support
for
sure
of
EPC,
as
Robin
mentioned
earlier.
B
Share
the
VPC,
enabling
is
a
requirement
for
Spotify
centralized
platform,
so
teams
can
run
jobs
to
access
services
in
our
projects.
Lastly,
there
was
no
option
to
upgrade.
In
our
case,
we
would
like
to
retain
certain
resources,
so
upgrading
cluster
would
not
lose
our
users
data
to
address
those
problems.
We
studied
using
deployment
command
line
tool,
KF
control,
to
build
our
cube
for
cluster.
B
However,
it
still
has
a
lot
of
problems.
First,
the
customization
was
hacky,
we
didn't
know
much
about
K
sonnet.
At
a
time
we
ended
up
with
manually
editing
many
settings
as
well
as
creating
our
own
case
on
deliver
file.
Those
manual
steps
are
not
ideal
and
cause
upgrading
and
creating
replicas
extremely
hard.
Moreover,
KF
control
doesn't
support
contacts,
so
it
was
very
error-prone
and
a
simple
human
mistake
could
destroy
the
cluster.
B
It
happened
before
once
got
to
change
the
contacts,
while
upgrade
a
cluster
and
I
accidentally
delete
entire
names
based
on
different
cluster,
so
tip
for
tip
of
today.
Added
a
contact
and
namespace
to
your
shell
command
prompts
it
helps
a
lot,
at
least
for
me.
On
the
other
hand,
Spotify
backhander
platform
team
is
using
terraform
to
manage
energy
K
clusters.
Those
clusters
are
also
configured
to
use
shared
VPSA,
so
why
don't
we
consult
the
kubernetes
experts
in
our
back-end
platform
team
and
also
use
care
forum
to
manage
our
DCP
resources.
B
Terraform
KF
control
is
no
longer
in
charge
of
creating
and
managing
tcp
resources.
Our
entire
instruction
stack
is
defining
a
care
forum
module
which
includes
at
gke
cluster
cross
eco
instance
for
metadata,
persistent
disks
service
accounts
and
a
lot
more
staff
care
forum
also
can
fix
your
VP
scenario:
settings
syncs
communities,
secrets
with
DCP
service
accounts
and
the
manager
database
database
users,
as
well
as
class,
object
rows
by
using
careful
module.
We
can
easily
replica
Kloster
for
different
environments
by
providing
a
different
set
of
parameters.
B
So
far,
we
have
seen
a
lot
of
benefits
of
using
terraform.
First,
there
are
many
existing
examples.
We
could
learn
from
back
and
platform
team
which
dramatically
lower
the
barrier
of
managing
our
platform.
Second,
it's
really
easy
to
reboot
modify
track
changes
to
our
instructure.
It
becomes
less
intimidating
now.
Third
cranial
replica
is
easy.
With
telephone
module,
we
use
it
to
beautify
clusters
for
our
platform,
one
for
experiments,
one
for
dev
and
a
worm
for
production.
B
B
So
when
we
make
changes
to
our
instructure,
we
open
a
PR
and
our
Beauty
system
can
plan
the
changes
ahead,
so
they
can
be
revealed
by
peers
before
being
applied.
If
the
beauty
system
detects
some
resource
is
going
to
be
deleted,
the
PR
will
be
automatically
blocked
at
Annie's
administrative
privilege
to
merge
it
because
we
believe
reverses
deletion
needs
extra
attention
to
avoid
any
mistakes.
B
At
these
points,
we
were
pretty
comfortable
of
using
case
on
it
after
a
KF
control
generator
or
the
kubernetes
resource.
We
use
the
case
on
e
to
self
parameters
for
different
environments
such
as
hostname
for
ingress
cloud
sequel
instance
and
persistent
disks
in
E,
and
we
in
stock
cube
flow
apps
through
KF
control
up
like
mount
very
soon,
just
after
we
must
a
case
on
it,
starting
from
version
0.6
cube
flow
uses
customized
as
a
default
Department
framework
since
K.
So
since
K
sonnet
project
has
reached
its
end
of
life,
that's
great.
B
With
customized
we
can
create
overlays
on
top
of
base
manifests
for
customer
department.
It
was
a
little
bit
overwhelming
to
us
in
the
beginning,
because,
instead
of
using
his
own,
it
command-line
tool
to
update
a
few
parameter
files.
Now
we
own
a
tons
of
yama
files,
but
the
process
is
similar.
We
first
uses
a
key
of
control
to
generate
manifests.
Then
we
create
our
own
overlays.
On
top
of
that
and
in
the
end,
K
fkf
control
apply
command
to
install
or
the
cube
flow
apps.
B
We
are
stealing
in
the
best
practice
of
using
customized.
We
hope
with
my
knowledge,
we
were
using
my
wisely
and
imagining
ID
award
tube
flow
deployment.
Changed
can
be
tracked
and
updates
in
the
components
can
be
easily
identified.
In
addition,
if
the
manifest
generated
by
Kiev
control
can
be
automatic
converted
to
our
own
customized
layouts,
then
we
could
use
Argos
ad
to
deploy
our
manifests
to
our
cluster
with
get
ups.
We
are
working
on
this
and
hopefully
it
will
be
make
our
deployment
much
easier.
B
B
Currently,
we
have
about
a
hundred
spotfires
using
a
platform
and
they
most
interact
with
cluster
in
two
ways:
Python
SDK
and
web
UI.
Python
SDK
is
an
internal
library.
We
build
to
help
to
have
a
people
to
develop
tf-x
workflow
to
run
with
cube
law
of
pipelines.
Web
UI
is
mostly
used
to
check
experimental
results
and
interact
with
pipeline
art
for
the
visualizations.
B
First
thing
we
need
to
make
sure
is
only
authenticated:
users
can
use
our
platform.
Luckily,
on
GCP
we
can
use
Google
our
ap
to
secure
our
web
UI
for
the
Python
SDK
access
is
managed
by
two
layers:
google.
I
am
a
science
user
project,
reviewer
robe
and
the
kubernetes
attack
grunts
user,
more
granular
permissions
to
access
specific
set
of
API
on
cluster.
B
The
other
thing
we
want
to
make
sure
is
that
teams
can
use
their
own
service
counts,
to
run
jobs
and
read
and
write
data,
usually
teams
and
Spotify
operating
in
different
GCP
projects.
We
start
service
counts
on
cluster
as
secrets,
so
platform
users
can
use
them
to
trigger
their
machining
work
flow.
We
know,
starving
service
counts
on
cars
is
non-ideal
and
we
look
forward
to
you
using
Google
workload.
Identity
in
the
future
after
multi-tenancy
is
a
party
is
implemented
in
the
cubes
or
pipelines.
B
B
B
Currently,
our
cluster
offers
three
predefined
ánotá
pools
when
Noda
pool
is
for
standard
emeritus
'kx
with
balance
the
CPU
and
memory
another
one
is
for
memory,
intensive
job
with
high
memory
machines
and
the
last
one
is
for
GPU
training.
When
users
write
them,
when
users
write
their
machine
learning
pipelines,
they
are
able
to
configure
resource
request
a
limit,
so
kubernetes
can
pick
the
right
in
order
to
launch
the
pod
and
release
a.
B
We
also
noticed
that,
where
more
and
more
users
are
using
our
platform,
the
D
for
the
resource
config
is
no
longer
sufficient
for
some
critical
service,
such
as
sto
parts,
a
sto
telemetry
and
a
metro
server.
We
need
to
manually
tune
those
resource
parameters
for
them
to
fit
our
classes
setups
and
a
usage
pattern.
For
this
part,
we
are
still
learning.
The
best
part
is
to
make
our
cluster
more
reliable.
B
We
launched
our
in
issue
version
of
the
platform
in
August.
So
far
we
have
our
they
see
about
hundred
users
using
it
and
15,000
runs
on
it.
You
can
see
from
graph
there
with
a
spike
in
the
middle.
That
was
a
renegade
job
running
every
five
minutes
to
stress
test
our
cluster.
It
took
us
a
while
to
detect
it
since
we
were
still
the
RFI
release,
but
the
great
news
is
that
it
was
running
successfully
and
we
even
didn't
notice
it.
So
in
the
future
we
definitely
would
like
to
have
more
monitoring
of
our
system.
B
With
our
centralized
cube
flow
platform,
ml
engineers
now
can
focus
on
designing
analyzing
ml
experiments
instead
of
beauty
and
maintaining
their
own
in
structure.
This
makes
a
model
iteration
cycle
shorter
and
it
results
in
faster
time
front
part
Iping
to
production.
Eventually,
we
can
have
better
Ahmir
models
in
our
products
and
make
our
user
happier.
B
B
B
Looking
back,
we
definitely
should
work
with
kubernetes
experts
fund
for
other
teams
as
early
as
possible,
because
4
ml
daily
engineers
kubernetes,
he
has
a
steep
learning
curve,
not
only
kubernetes
beauty.
A
centralized
mail
perform
also
involved
knowledge
around
yellow
working
security
deployment,
resource
management,
a
lot
more
stuff.
We
need
to
collaborate
with
experts
from
other
domains,
so
it's
a
teamwork.
B
Second
cube
flow
is
too
big
to
chew,
or
at
once
it
has
so
many
components,
and
each
of
them
is
very
complicated.
So
you
need
to
be
patient
and
understand.
I
use
them
piece
by
piece
to
fit
your
own
needs.
This
is
one
of
the
reason
we
choose
to
build
a
centralized
perform,
so
we
can
centralize
this
knowledge
in
infer
team
and
they
let
ml
engineers
focus
on
building
products
using
our
platform.
B
B
That's
all
for
our
talk,
beauty
and
managing
our
centralized,
the
cube
flow
platform
and
Spotify.
If
this
sounds
exciting
to
you,
come
join
us
in
our
awesome
office
in
New,
York,
Stockholm,
Boston
or
London.
Since,
for
a
time
we
will
get
to
show
off
what
we
have
been
working
on
happy
to
take
any
questions
now.
A
B
So
another
question
is:
does
every
developer
has
their
own
namespace
in
terms
of
customization
for
gke
I?
Think
we
have
our
own
DNS
server
for
our
cheek
a
year
and
we
also
have
shared
VPS
a
setup
for
cheek
a
cluster.
That's
the
true
like
may
saying
we
have
done
for
our
cluster
right
now,
since
the
queue
flow
pipeline
doesn't
have
the
match
he
uses
apart.
So
everybody
is
using
the
same
namespace,
but
very
soon
they
gonna
added
much
user
tendencies
apart.
They
are
we
so
different
teams
can
have
their
own
namespace.
B
Okay,
so
the
question
is:
how
do
we
manager
the
GPU
Porter
resource
you
right?
To
be
honest,
there
are
not
many
people
using
GPU
at
a
Spotify,
the
most
of
machine
learning
tasks
as
Spotify
is
for
recommendations
so
like
we
did,
it
do
any
special
staff.
It's
just
also.
The
cost
is
not
a
main
concern
at
this
point
for
us,
so
we
just
offer
everybody
if
they
want
to
use
the
they
can
use
anything.
A
A
B
B
Okay,
so
the
question
is
the
problems
we
are
having
migrating
from
customized
to
our
own
get-ups
right
and
so
the
one
major
pain
point
we
have
is
so
in
order
to
make
it
opsworks,
we
need
to
structure
order,
manifests
in
a
specific
way,
so
the
tip-offs
can
apply
them
to
customize,
but
a
KF
control
when
we
use
K
of
control
to
children.
So
those
are
the
manifests,
it
has
its
own
file
structure,
so
we
need
to,
like
him,
have
some
process
to
convert
the
structure
generated
by
K
of
control
to
our
own
structure.
B
B
A
Yeah,
so
the
question
is:
do
we
run
pipelines
everyday
or
when
we
get
new
data
in
terms
of
the
coop
flow
setup?
There
are
people
running
pipelines
all
the
time
as
sort
of
that
graph
showed
in
terms
of
like
machine
learning
at
Spotify,
there
are
people
who
do
daily
retraining.
There
are
teams
that
don't
do
daily
retraining.
A
Historically,
we
were
using
this
tool
that
people
may
have
heard
of
called
Luigi.
That
was
for
mostly
designed
for
our
data
pipelines
that
do
like
daily
or
perhaps
hourly,
if
you
want
retraining
or
re
like
running
a
pipeline.
So
a
lot
of
people
historically
use
that
to
do
that
to
schedule
their
training
and
depend
on
the
upstream
pipelines
that
were
that,
create
the
data
that
they
depend
on.