►
Description
Don’t miss out! Join us at our next event: KubeCon + CloudNativeCon Europe 2022 in Valencia, Spain from May 17-20. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Scaling Kubeflow for Multi-tenancy at Spotify - Keshi Dai & Jonathan Jin, Spotify
Spotify began offering a centralized Kubeflow Pipelines product to its machine learning teams around two years ago. Since then, adoption has skyrocketed, with more teams training more models and running increasingly complex experiments. These increased demands on our system come with more stringent demands on us, the Kubeflow team at Spotify, to ensure not just cluster reliability, but cluster equitability. Our job is to not just be cluster maintainers, but cluster stewards—ensuring equitable and reliable access to cluster resources, and keeping users from stepping on each others’ toes. In this talk, we’ll discuss our streamlined tooling to maintain, deploy, and monitor Spotify’s distribution of Kubeflow. We’ll illustrate the challenges we face as we scale to increased user load and increasingly distinct and demanding pipelines, and outline our approach to addressing those challenges with “multi-cluster” Kubeflow. Finally, we’ll give a preview of our future plans for the platform.
A
Let's
start
with
the
introduction,
my
name
is
cash
dai,
I'm
a
senior
ml
info
engineer
at
spotify.
This
is
my
teammate
jonathan,
jean
also
a
senior
engineer.
You
will
hear
from
him
during
the
later
part
of
this
talk,
so
we
are
spotify.
If
you
haven't
heard
of
us
before
we
an
audio
streaming
service,
we
launched
in
2008,
and
now
we
have
more
than
365
million
monthly
active
users
on
our
platform.
We
have
more
than
70
million
tracks
and
almost
3
million
podcast
titles
and
our
service
is
available
in
178
markets
all
over
the
world.
A
It
enables
us
to
recommend
artists,
playlists
and
podcasts
to
keep
our
users
active,
engaged
and
more
likely
to
subscribe
in
the
long
term
to
power
ml
products
at
spotify.
Our
team
is
building
a
standardized
machine
learning
platform
to
provide
our
engineers
with
tools
and
environment,
to
quickly
prototype
experiment
and
productionize,
their
ml
ideas.
A
A
This
is
a
typical
ml,
workflow
and
spotify.
As
you
can
see,
it
has
a
sequence
of
components
representing
different
steps
in
a
machine
learning
pipeline.
It
starts
with
future
engineering,
future
collector
and
transform
components,
assemble
the
raw
features
and
transform
them
into
the
meaningful
ones
for
model
training.
A
Then
we
have
the
trainer
component
for
model
training
and
evaluate
component
for
the
model
performance
analysis
in
the
end,
hopefully,
everything
looks
good.
Then
we
can
deploy
the
model
to
production
through
the
deployer,
but
of
course,
in
reality,
an
actual
ml
pipeline
can
be
much
more
complicated
than
this
example.
A
It
also
includes
user
custom
components
containing
different
business
logics
after
pipeline
is
authored,
teams
can
submit
pipeline
to
our
gk
cluster
and
it
will
get
executed
through
keyflow
pipelines
on
our
cluster.
We
installed
sdo
for
service
discovery
and
authentication,
and
now
metadata
start
to
track
tfx
component
executions
and
our
own
toolings
for
team
management
and
metric
monitoring.
A
A
A
Our
qr4
clusters
are
built
through
the
same
process.
The
gtp
resources
are
created
and
managed
by
the
terraform,
including
gk
clusters,
cloud
sql
instances,
service
accounts
and
more
terraform
is
a
tool
for
building
changing
and
versioning
the
infrastructure,
so
the
infrastructure
can
be
traded
as
code.
A
We
first
launched
our
platform
in
for
2019
and
we
had
a
beta
release
in
2020.
We
will
go
ga
in
early
next
year.
So
far
we
have
more
than
60
teams
and
almost
600
users.
Using
our
platform,
there
were
thirty
thousand
models
trained
and
a
hundred
thousand
pipeline
execution
hours
during
our
platform
last
year,
in
average,
there
are
close
300
pipeline
runs
every
day.
A
It
has
been
a
great
journey
for
us
to
witness
our
platform's
growth
and
be
part
of
it
in
the
following
section,
jonathan
and
I
are
going
to
talk
about
the
challenges
we
are
facing.
While
we
scale
up
our
platform
and
how
we
address
those
challenges,
more
specifically,
we
are
going
to
cover.
How
do
we
support
a
growing
number
of
ml
teams
on
our
platform
to
allow
them
to
operate
in
isolated
and
self-manageable
environment?
A
A
A
A
A
A
With
this
in
mind,
we
have
developed
our
own
team
management
tooling,
for
kubeflow
cluster.
In
our
setup,
each
team
profile
or
namespace
is
owned
by
a
google
group
that
corresponds
to
an
internal
ldap
group,
which
removes
the
need
of
an
individual
user
being
the
owner
to
support
the
team
concept
in
the
kubeflow
profile.
A
We
also
created
a
corresponding
kubernetes
controller
to
support
google
group,
so
it
automatically
extracts
members
from
the
group
and
configures
necessary
permissions
for
them.
The
controller
also
periodically
syncs
with
google
group
service,
so
any
changes
in
the
team
will
be
automatically
reflect
on
cluster.
A
Lastly,
we
use
customize
to
manage
our
team
configs
and
deploy
them
through
githubs.
An
engineer
in
a
team
can
submit
a
pull
request
to
repo
where
all
the
profiles
are
stored,
then
the
cd
runs
customize
and
verify
the
changes
once
the
pi
is
merged.
The
master
building
invokes
customize
to
render
resources
and
deploy
them
to
the
clusters.
A
A
A
The
profile
also
requires
gcp
service
account
to
access
datasets
and
gcp
resources
from
its
namespace.
It's
bounded
with
a
kubernetes
service
account
through
gke
workload,
identity,
contribute
field
defines
a
list
of
additional
users.
The
team
would
like
to
give
access
to.
They
can
be
individuals
or
groups.
A
After
team
configuration
is
defined,
our
tool
automatically
converts
it
into
a
set
of
kubernetes
resource
files.
As
you
can
see
on
the
right
side,
it
has
a
cube
flow
profile
yaml
and
the
example
namespace
folder
in
the
namespace
folder.
There
are
crd
resource
file
that
defines
outback
and
sto
rules
for
team
members
and
contributors.
A
A
A
We
also
have
a
dev
cluster
intended
for
platform
developers,
but
it
also
serves
as
backup
cluster
when
the
product
is
under
upgrade
or
maintainers.
This
setup
makes
rolling
out
a
new
version
of
queue,
flow,
painful
and
slow.
We
usually
first
test
it
internally
on
our
experimental
cluster,
then
apply
the
changes
to
the
dev
and
then
wait
for
our
users
to
upgrade
the
pipeline
before
we
could
eventually
roll
it
out
to
the
proud.
The
process
usually
takes
weeks
to
complete
to
support
the
roll
out.
A
A
A
The
pipeline
is
guaranteed
to
be
able
to
run
on
a
cluster
with
the
same
major
version
of
the
sdks
that
views
it:
the
cluster
installs
seller
of
services,
with
a
version
compatible
with
each
other,
as
well
as
client
sdk.
As
shown
in
the
illustration
suppose,
we
are
committed
to
support
three
major
versions.
A
A
So
why
is
this
better?
It
first
encapsulates
the
entire
q
flow
stack
from
the
pipeline
sdk
to
the
execution
engine
to
ensure
compatibility.
Second,
it
decouples
the
instructor
upgrade
cadence
from
the
pipeline,
upgrade
cadence
users
can
upgrade
their
pipelines
based
on
their
own
priorities
and
schedules.
A
A
Last
but
not
least,
we
will
no
longer
need
to
worry
about
breaking
changes
and
backward
compatibility.
It
can
be
simply
addressed
by
offering
users
a
new
cluster,
but
what
does
that
mean
for
our
team?
This
multi-class
strategy
definitely
requires
us
to
have
more
sophisticated
setup
for
our
cluster
management,
so
we
consolidated
our
terraform
process
to
encapsulate
the
entire
cluster
creation
logic
in
one
module,
so
different
clusters
can
be
easily
critical.
A
Finally,
this
is
illustration
of
our
multi-cluster-based
q4
platform.
Each
qp4
cluster
is
built
through
our
queue
flow
deployment
blueprints.
On
top
of
that,
we
have
a
centralized
ml,
workflow
manager
service
and
a
unified
ui
for
pipeline
experiments
and
runs
next
I'll
hand.
Over
the
talk
to
my
teammates
jonathan,
he
will
talk
more
in
details
about
how
we
improve
the
queue
flow
deployment
process
for
our
new
multi-class
setup.
A
B
A
B
B
In
response
to
all
these
new
and
amplified
challenges,
we
decided
to
adopt
argo
cd
as
the
framework
around
which
to
base
our
cluster
deployment
operations
moving
forward.
This
is
a
natural
extension
of
our
existing
github
setup
now,
instead
of
applying
all
of
our
changes
by
hand
to
each
user
facing
cluster
that
we
oversee
and
maintaining
a
loose
mental
model
of
what
dependencies
exist
between
what
resources
and
what
follow-up
steps
need
to
be
taken
to
bootstrap,
we.
B
B
This
streamlines
the
deployment
process
pretty
significantly
and
it
allows
our
team
to
focus
on
higher
level
requirements
while
offloading
that
rote
mechanical
work
of
propagating
these
changes
to
argo
cd
more
critically.
In
doing
so,
we
ensure
greater
reliability
for
users
of
this
product
and
less
chance
of
breakages
or
inconsistencies
as
a
result.
B
A
B
Insight
that
our
team
came
to
here
is
that
users
are
not
alerts
and
they
should
not
act
as
an
alerting
system.
What
I
mean
by
this
is
that
our
users
should
not
be
the
first
to
know
about
any
issues
in
our
clusters
and,
by
extension,
we
as
cluster
owners
and
maintainers,
should
know
about
those
issues
before
anybody
else.
In
other
words,
if
a
user
has
been
impacted
by
an
outage
or
an
inefficiency
or
a
regression,
it's
too
late,
basically,
and
with
multi-cluster,
we
now
have
more
complex
of
observability
needs.
B
B
To
that
effect,
we
took
further
advantage
of
the
infrastructure's
code
paradigm,
with
our
terraform
setup.
More
fundamentally,
however,
we
expanded
our
notion
of
infrastructure
and
what
that
really
entails.
At
first
glance,
one
might
consider
infrastructure
as
solely
the
compute
resources
and
the
networking,
for
example,
that's
needed
to
run
your
cluster
and
its
constituent
services,
but
we
argue
that
really
it's
it's
so
much
more
than
that.
It.
B
A
A
B
Auxiliary
deployments
or
sidecars
that
you
might
need
in
order
to
supplement
the
core
parts
of
your
product
offering
so
with
that
going
back
to
our
terraform
setup
for
a
second
on
top
of
using
terraform
to
configure
and
provision
compute
resources,
node
pools,
database
instances
and
similar
other
more
we'll
call
it
concrete
infrastructural
requirements.
We
can
extend
that
configurability
to
standardized
dashboards,
alert
policies
and
formalized
slos,
all
tracked
within
gcp
and
all
configured
using
the
official
gcp
terraform
provider.
B
A
B
Can
now
stamp
out
new
clusters
much
like
creating
new
instances
of
a
class
or
a
struct
in
your
typical
programming
language,
but
now
they
also
come
with
batteries
included
essentially,
and
all
the
requisite
tooling,
on
top
of
the
raw
compute
comes
for
free
as
well.
This
entire
pattern
allows
us
to
spin
up
and
tear
down
entire
cube
load
clusters
with
ease,
and
all
this
with
ironclad
guarantees
that
everything
is
set
up
for
each
cluster,
exactly
the
way
that
it
should
be,
and
we
expect
it
to
speaking
of
observability.
B
However,
let's
talk
a
little
bit
more
specifically
about
slo
tracking
for
our
kubeflow
platform
as
the
platform
matures,
we
have
an
increased
need
for
visibility
with
regards
to
performance,
stability
and
reliability.
This
applies
not
just
to
us
as
cluster
owners,
but
to
our
users
as
well.
They
want
to
know,
and
really
they
deserve
to
know
what
kind
of
performance
they
can
expect
from
our
from
our
clusters.
If
they're
going
to
adopt
the
adopt
use
themselves
like.
B
A
B
And
whistles
for
slo
tracking,
for
example,
we
can
track
what's
called
at
the
error
budget
and
the
burn
rate,
which
gives
us
ready
insight
into
how
close
we
are
at
any
given
time
to
violating
our
slo.
We
can
also
define
alerts
on
top
of
the
error
budget
and
the
burn
rate
such
that,
if
we're
starting
to
run
low
on
error
budget,
for
example,
we
can
take
appropriate
remediations
to
stay
within
budget
at
for
that
time
frame
and
prevent
ourselves
in
advance
from
violating
our
own
slows.
B
A
B
Provided
by
our
respective
dependencies,
including
istio,
qflo
and
also
kubernetes
itself,
via
cube
state
metrics,
however,
we
noticed
pretty
early
on
that
a
lot
of
these
metrics
in
trying
to
be
as
general
as
possible
and
avoid
domain
specificity
often
don't
outright
enable
us
to
track
specifically
what
matters
to
us
and
what
is
critical
to
our
product
offering
and
even
then
cases
where
it
was
in
fact
possible
to
do
so.
Oftentimes
required
these
really
lengthy,
really
arcane,
really
verbose,
recording
rules
and
extensive
prom,
ql
arithmetic
to.
B
Maintainability
became
an
issue
to
that
effect.
We
implemented
what
we
call
very
appropriately
kubeflow
state
metrics.
We
took
heavy
inspiration
from
coop
state
metrics
to
create
an
analogous
custom
metrics
exporter,
specifically
targeting
keyflow
use
cases,
specifically
like
q
state
metrics
when
deployed
to
our
clusters.
Kubeflow
state
metrics
will
listen
on
kubernetes
events
such
as
pod
creation
or
scheduling
changes
and
translate
those
into
appropriate
product
specific,
prometheus
metrics.
These
might
include
tracking
how
long
kubeflow
pipeline
pods
take
to
start
running
how
long
those
pods
stay
on
the
cluster
once
execution
concludes
and
so
on.
B
With
all
that,
I
want
to
talk
a
little
now
about
some
of
the
key
lessons
that
our
team
has
learned
throughout
all
this
work.
The
first
lesson-
and
this
is
one
that
I
touched
on
before-
is
that
your
infrastructure
is
more
than
just
compute.
In
particular,
it
is,
in
our
opinion,
worth
opening
yourself
to
thinking
about
your
cluster
infrastructure
in
more
abstract
areas.
B
B
And
time
consuming
to
put
in
place
reactively
after
the
fact
and
all
the
while
users
are
being
frustrated
by
outages,
confusing
behavior
and
continuing
to
lose
confidence
in
your
platform
over
time.
And
lastly,
I
want
to
encourage
expanding
your
notion
of
what
your
product
truly
is
taking
the
spotify
qflo
platform.
As
an
example,
we
argue
that
our
platform,
while
it's
based
primarily
around
qflo,
is
not
exclusively
kubeflow
itself.
Really.
B
What
we've
done
is
treat
kubeflow
as
an
extensible
foundation
that
we
molded
customized,
extend
it
and
continue
to
do
so
really
to
fit
our
ever-growing
platform
needs,
and
these
include
all
the
new
functionality
that
we
described,
such
as
multi-tenancy
multi-cluster,
support,
reliability,
instrumentation
and
custom.
Metrics.
B
A
B
We
also
want
to
invest
more
in
users
ability
to
manage
the
particulars
of
their
key
flow
journey
themselves.
Basically,
this
might
include
management
of
service
accounts,
permissions
retention
policies
and
cluster
quota
overrides
to
allow
them
to
temporarily
take
up
or
like
consume
more
resources
at
a
time
for
intensive
compute.
B
Doing
so
would
give
advanced
users
greater
control
over
their
own
platform
usage,
while
still
allowing
newcomers
to
get
quickly
get
started
with
sensible
defaults
that
we
provide
for
them
and,
lastly,
we
believe
very
strongly
in
contributing
back
to
the
open
source
community
that
has
empowered
so
much
of
our
own
work
case
in
point.
A
lot
of
our
work,
such
as
the
kupelo
centric
metrics
exporter,
that
I
mentioned
before,
has.
B
Looking
at
ways
to
open
source,
some
of
our
work
around
kubeflow
in
production,
details
are
kind
of
fuzzy
right
now,
but
there
will
hopefully
be
more
to
come
and
we'd
love
to
share
more
with
you
in
the
near
future,
and
that's
all
we
had
to
talk
to
you
about
today,
thanks
a
bunch
for
listening
to
us
today.
If
any
of
this
work
sounds
interesting
to
you,
we
welcome
you
to
check
out
life
spotify.com
for
new
opportunities,
around
kubernetes
cloud
native
computing
and
machine
learning
at
spotify.
Thanks
again
for
your
time,
we'll
be
taking
questions
now,.