►
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Making an Internal Kubernetes Offering Generally Available - James Wen, Spotify
In the span of two years, Spotify went from two developers investigating what a potential migration to Kubernetes might involve to having an internal, multi-tenant offering of Kubernetes become generally available for all its developers as the new, primary runtime offering. Spotify has previously given talks on the earlier bootstrapping, experimentation, alpha, and beta phases of this migration process. However, this talk will focus on the latter work involved in bringing the internal offering of Kubernetes “across the finish line.” The talk will cover what was required to bring the offering to general availability, including work shoring up scalability and reliability via a multicluster strategy, DIRT testing, operational metrics and alerts. This talk will also cover the technical and process elements involved in designing a successful self-service migration experience for developers.
https://sched.co/Uab1
A
Hi
I'm
James
1
and
from
Spotify
and
I'll
be
talking
about
making
an
internal
communities
offering
generally
available
so
we're
currently
in
the
process
of
migrating
our
runtime
over
to
kubernetes.
It's
modify
and
it
only
took
about
two
years
to
go
from
just
two
developers
investigating
the
potential
usage
of
criminais
DS.
That's
modify
to
going
to
having
a
full
GA
generally
available
offering
for
all
of
our
developers,
and
so
we've
given
past
talks
about
the
first
few
migration
phases.
A
But
this
talk
aims
to
share
on
kind
of
the
latter
parts
and
what
it
went
into
bringing
our
GA
offering
over
like
the
finish
line.
In
a
sense,
it's
quick
about
me.
I'm,
a
senior
at
Surry
at
Spotify
I,
started
out
on
a
team
that
worked
on
provisioning
and
DNS
and
now
within
the
same
core
infrastructure
organization,
I'm
on
a
team
that
maintains
deployment
and
is
one
of
the
teams
responsible
for
our
kubernetes
migration.
A
So,
in
terms
of
the
agenda,
I'll
talk
about
briefly
introduce
that
Spotify,
how
we
do
operations
and
what
our
runtime
looks
like,
what
we're
migrating
from
to
kind
of
seed,
the
context
there
and
then
we'll
dive
into
the
GA
journey
and
cover
some
learnings
and
takeaways.
So
very
quick
about
Spotify
we're
an
audio
streaming
platform
across
a
variety
of
different
devices,
and
we
aim
to
connect
millions
of
artists
with
billions
of
fans.
A
In
doing
so,
we
have
a
fair
amount
of
users
and
subscribers
who
listen
to
a
fair
amount
of
audio
content,
and
all
of
this
is
supported
by
a
large
amount
of
Engineers
across
a
large
amount
of
teams
that
are
maintaining
a
lot
of
micro
services
that
get
a
fair
amount
of
load
and
generate
a
fair
amount
of
data
and
events.
So
how
are
these
engineers
doing
operations
for
these
applications
and
micro
services?
A
So
if
we
have
like
an
operational
spectrum,
where
say
on
one
side,
you
have
teams
that
have
complete
team
autonomy
and
how
they
do
their
operations.
They
can
deploy
or
run
stuff
set
up
metrics,
using
whatever
tools
or
processes
they
want,
and
then,
on
the
other
side
of
the
spectrum,
you
might
have
like
a
centralized
ops
team
that
is
responsible
for
all
the
DevOps
relay
things
for
your
product
and
is
responsible
for,
like
ultimately,
your
product
being
up
and
running
Spotify
kind
of
operates
in
the
middle,
with
like
an
ops
and
teams
model.
A
So
here
teams
build
their
own
services.
They
are
operating
them
as
well,
but
they're
doing
so,
using
pretty
extensive,
tooling
and
like
a
platform
that
the
court
infrastructure
organization
provides
for
them,
so
they're
still
doing
operations
but
they're
doing
it
at
a
much
higher
abstraction
level,
and
a
great
actualization
of
this
is
something
we
call
the
golden
path.
So
this
is
an
opinionated
blessed
like
set
of
best
practices
for
building
and
operating
services
at
Spotify,
which
includes
a
pretty
comprehensive
tutorial.
A
That
developers
can
follow
on,
like
the
very
first
day
to
build
the
new
service,
like
put
it
into
production
and
set
up
things
like
metrics,
alerting
on
call
for
it.
So
let's
talk
abou
quickly
about
our
old
runtime
or
what
we're
migrating
from.
So
our
old
runtime
involves,
having
like
one
docker
container
or,
like
instance,
of
a
micro
service
per
host
and
the
host
configuration
is
set
up,
is
done
by
puppet
and
the
docker
orchestration
is
done
by
Helios,
which
is
our
own
open
source
container
orchestration
system.
That
was
conveniently
conveniently
announced
the
day
before.
A
Kubernetes
was
so
we're.
Moving
from
that
old
runtime
to
our
new
runtime
kubernetes
or
more
specifically,
like
gke,
and
with
our
offering
we
with
kubernetes,
we
no
longer
need
helios
or
puppet
the
concerns
and
like
management
around
docker
and
like
GCE
hosts
are
now
offloaded
to
gke,
which
lets
has
let
us
build
a
much
more
advanced
like
criminales,
offering
for
developers
on
top
of
it,
that
is
using
soft
multi-tenancy
and
is
a
multi
cluster
internal
offering
and
I.
Think.
A
The
last
piece
of
like
important
pre
context
is
that
our
goal
for
this
migration
has
been
scoped
to
running
all
stateless
back-end
services
at
Spotify,
on
kubernetes
and,
if
you're
interested
in
some
like
the
earlier
migration
phases,
because
this
definitely
wasn't
like
a
big
monolithic
migration
effort.
You
can
see
some
previous
talks
we've
given
on
that,
so
general
general
availability
for
our
offering
like
what
does
it
actually
mean?
How
did
we
define
it?
A
And
so
this
has
been
pretty
successful
so
far,
but
if
I
do
say
so,
and
even
though
we're
not
currently
like
actively
pushing
or
forcing
our
developers
to
move
over
to
communities,
we've
seen
that
roughly
33%
of
eligible
stateless
back-end
services
have
moved
over
to
kubernetes.
So,
to
put
that
in
perspective,
like
all
of
these,
migrations
were
done
by
teams
that
did
so
through
the
self-service
experience,
and
they
did
so
on
their
own
of
their
own
volition,
like
on
their
own
time.
A
So
now,
let's
get
into
what
we
had
to
do
to
actually
make
the
offering
ready
for
a
general
availability
these
break
down
into
like
a
few
big
domains,
the
first
being
operational
solidity
kind
of
like
scalability
reliability.
Observability
first
topic
I
want
to
talk
about
here-
is
our
multi
cluster
strategy
and
how
we
manage
the
clusters.
A
So
we
started
out
with
three
just
three
product
production
clusters,
but
now
we're
running
about
like
2,500,
fairly
large
nodes
across
15
production
clusters
that
are
across
like
three
different
regions,
and
so
in
the
beginning
we
had
automated
scripts
and
they
were
pretty
quick
to
create
quick
to
change
easy
to
run.
That
did
all
of
this
like
cluster
creation
and
manage,
but
as
we
scaled
up
with
more
clusters
and
those
clusters
got
more
complex,
we
started
to
encounter
a
few
issues.
A
For
example,
these
scripts
were
not
idempotent
in
the
case
of
like
this
script,
failing
out
in
the
middle
of
it
and
additionally,
the
creation
part
of
the
scripting,
which
I
would
say,
is
the
most
important
part.
Actually
that
wasn't
something
that
got
tested
or
executed
that
often
again
in
practice,
because
we
weren't
creating
new
clusters,
all
that
often
yeah,
and
so
we
moved
to
a
declarative
approach
with
terraform
and
if
you're
interested
in
some
like
deeper
details
on
our
experiences
there.
A
My
colleague
David
gave
a
really
great
keynote
on
that
last
year,
at
Cuba
Khan
and
the
takeaway
for
like
the
multi
cluster
strategy,
and
our
management
of
it
is,
if
it's
possible
offload.
The
cluster
management
we
realized
is
that
somebody
has
to
be
like
consistently
concerned
with
making
sure
that
cluster
creation
bootstrapping
is
like
up
to
date.
Work
as
expected.
That
kind
of
thing-
and
it's
really
great-
to
be
able
to
offload
this
to
like
an
open
source
tool
like
terraform
or
like
a
managed
solution
or
like
a
provider
of
some
sort.
A
The
other
aspect
of
multi
cluster
are
multi.
Class
strategy
was
deployment.
How
is
the
deployment
to
multiple
sets
of
clusters
occurring,
and
so
here
we
were
able
to
leverage
an
existing
deployment
system
that
we
had
called
tugboat,
which
served
as
our
like
deployment,
abstraction
layer,
our
for
developers,
and
so
this
existed
before.
I
moved
to
chrome
Nettie's,
the
kind
of
left
branch
there
already
existed
and
was
how
people
used
tugboat
to
deploy
their
services.
A
Dude
helios,
namely
their
c
ICD
pipelines,
would
basically
talk
to
tugboat
and
talk
about,
would
orchestrate
the
deploy
out
to
GCE,
and
so
we
kind
of
hid
or
built
our
deployment
to
kubernetes
behind
this
existing
abstraction.
That's
that
right
branch
right
there
and
how
multi
cluster
plays
into
this
is
that
we
built
the
service
called
compass,
fairly
simple
service,
that
we'll
talk
to
two
basically
requests.
A
So
this
is
proven
to
be
pretty
true
and
what
we
wanted
to
do
on
the
deployment
tooling
layer
was
that
just
ensure
that
their
app
is
on
some
cluster
in
the
region
that
they
want
to
be
in
I'm,
just
interested
in
some
I'm
plugging
a
lot
of
different
talks
here.
But
if
you
were
interested
in
some
of
the
deeper
details
around
like
our
multi
cluster
strategy
and
also
some
things
around
scalability
with
running
large
clusters,
my
colleague,
Austin
and
Iago
Jago
from
Google
or
giving
a
talk
on
this
tomorrow,
a
cube
con.
A
So
let's
move
on
to
operational,
metrics
and
alerts.
So
this
is
for
us
as
the
owners
of
the
cluster,
not
specifically
for
like
application
developers
using
the
clusters,
so
we
had
to
set
up
pretty
solid
monitoring
on,
like
integration,
components,
error,
latency
saturation
on
these
components.
So
what
do
I
mean
by
integration
components?
These
were
things
that
we,
the
cluster
owners,
were
running
on
the
clusters
or
interfacing
with
the
clusters
that
provided
integration
with
our
like
existing
infrastructure,
for
example,
things
like
service
discovery,
IT,
general
controls
and
our
in-house
secret
system.
A
So
we
wanted
to
monitor
the
monitor
those
pretty
closely
ensure
that
there
was
integrations
like
I've
kept
working.
We
also
had
a
pretty
important
smoke
tests
metric
that
would
basically
periodic
periodically
deploy
a
test
app
to
our
clusters,
and
this
was
less
about
the
cluster
health
and
more
about
the
deployment
tool
chain
and
making
sure
that
that
was
working
end
to
end.
A
They
made
sure
that
we
had
the
proper
metrics
and
alerts
set
up
to
be
able
to
be
aware
of
the
situation
like
during
these
dirt
tests,
dirts
and
also
it
validated
that
our
playbooks
were
working
as
expected,
specifically
around
things
like
cluster
creation,
backup
restoration
and
it
validated
that
our
backups
were
actually
working.
Because
during
the
cluster
deletions,
we
had
to
restore
resources
that
we
had
backed
up
because
as
they
say,
if
you
don't
have,
if
you
don't
test
restoring
your
backups,
you
don't
actually
have
backups.
A
So
this
ensured
that
we
actually
had
backups
and
the
take
over
here
is
to
plan
for
and
simulate
failures
before
general
availability,
whether
you're
operating
your
own,
like
kubernetes
clusters
or
you're.
Using
a
cloud
manage
like
offering
of
some
kind
I,
don't
make
the
assumption
that
the
clusters
will
always
be
there
and
be
stable,
have
a
plan
and
processes
in
place
for
when
this
assumption
is
broken.
A
A
A
This
is
a
graph
of
our
cost
over
time
of
micro
services
that
are
currently
fully
migrated
over
from
helios
to
helios
on
GCE
to
our
kubernetes,
offering
on
gke.
So
a
quick
explanation
of
what
you're,
seeing
here
in
our
old
runtime,
with
like
the
one
service
instance
per
host,
we
didn't
have
been
packing
and
we
had
some
limitations
around
auto-scaling.
A
So
we
had
this
gke
onboard
dr
container,
that
we
supplied
to
users
that
they
could
run
like
locally
on
their
machines,
and
this
would
basically
generate
the
proper
like
namespace
related
resources
for
their
team
or
like
desired
namespace
on
a
mono
repo
called
system
resource
manifests
so
to
explain
this
our
resource
manifest
like
distribution
model
and
is
that
the
resource
manifests
or
resources
that
we
want
applied
to
all
clusters.
So,
for
example,
things
like
namespace
information
things
like
integrations
that
we
as
the
cluster
owners,
operate
these
go
into
this
SRM
system.
A
Resource,
manifests,
mono
repo
and
get
applied
to
all
clusters
and
then
for
people's
like
resource
manifests
that
are
like
related
to
their
application.
Those
live
with
the
actual
application
repos.
So
there's
really
just
like
one
step
for
a
user
or
like
a
team
to
run
to
get
their
own
namespace
and
the
takeaway.
There
is
to
not
be
afraid
of
taking
complexity
for
your
developers,
I'm.
Speaking
from
the
standpoint
of
like
being
a
person
in
infrastructure
who,
like
owns
the
kubernetes,
offering
I,
think
it's
always
said.
A
Things,
like
you
know,
keep
it
simple,
but
you
shouldn't
necessarily
like
keep
your
design
of
the
offering
like
behind
the
scenes
simple.
If
that
introduces
extra
complexity
for
your
developers,
like,
don't
be
afraid
of
making
things
a
little
more
complex
under
the
hood,
so
that,
for
our
case,
developers
only
have
to
run
a
single
doctor
container
to
like
get
their
namespace
set
up
and
ready.
A
The
next
next
aspect
is
around
applications
and
like
how
so
they
have
their
name
space
like
how
are
they
actually
getting
their
applications
there.
So
this
shows
what
the
workflow
was
before
kubernetes
the
like
kind
of
dotted
box.
There
is
what
is
in
their
application
repo,
and
here
it
is
a
Helios
configuration
file
that
they're
the
C
ICD
pipelines
basically
run.
These
two
commands
related
to
tugboat
on
to
create
a
new
version
and
then
set
that
as
the
desired
like
target
version
and
that
causes
tugboat
to
actually
deploy
out
to
GCE
and
Helios.
A
So,
with
the
kubernetes
experience
or
say
migrating
from
that
to
running
on
both
Helios
and
kubernetes,
all
developers
had
to
do
was
basically
run
this
cookie
cutter
tool
that
we
provided
them
to
get
like
a
working
set
of
manifests
for
their
applique
and
then
also
add
this,
like
yellow
file
shown
on
the
right
called
like
deployment
info
dot.
Llamo
that
specifies,
in
this
case,
I
want
to
deploy
to
both
Helios
and
kubernetes,
and
so
the
CI
CD
pipeline
part
stays
exactly
the
same.
A
It's
the
same
exact
two
commands
talked
about
commands
that
are
being
run
to
then
deploy
to
Helios
and
gke
criminales,
so
that
the
top
to
the
top
right
boxes
are
like
all,
that's
new
and
the,
and
then,
after
that,
if
they
want
to
remove
the
Helios
capacity
or,
if
they're,
starting
out,
like
with
the
say,
Golden,
Path
and
they're
only
on
kubernetes.
That's
what
this
would
look
like
and
I
think.
A
The
thing
I
want
to
point
out
is
that
in
this
deployment
info
mo,
they
can
also
specify,
like
very
high
level
deployment
or
like
kubernetes
related
configuration,
for
example
like
the
region's
that
they
want
to
be.
In
things
around
like
region,
redundancy
with
like
clusters
per
region,
things
like
whether
you
want
a
canary
also
goes
into
this
file,
and
our
takeaway
here
was
to
use
existing
tools
and
abstractions
wherever
possible,
to
make
it
as
frictionless
as
possible
for
your
developers
to
migrate
over
themselves.
A
So
like
tugboat
tugboat
was
deployment
for
most
of
our
developers
from
both
like
a
tooling
and
like
an
abstraction
level
kind
of
perspective.
So
we
hidden
build
the
kubernetes
deploying
process
behind
this
existing
abstraction
and
because
of
that,
like
the
CIC,
via
pipelines,
stayed
exactly
the
same
and
the
overall
deployment
workflows
that
developers
were
used
to
stayed
the
same
and
how
they
conceptualize
deployment
stayed
the
same
for
the
most
part.
A
The
next
aspect
is
around
documentation,
everyone's
favorite,
which
has
become
our
favorite
as
well.
We
wrote
an
extensive
amount
of
internal
documentation,
and
you
know
these
are
always
like
workin
Proctor,
work-in-progress,
you're,
always
iterating
on
them
and
are
like
I
think
use.
We
put
that
on
to
stack.
Overflow
and
prai's
are
internal,
like
stackoverflow
enterprise,
in
the
form
of
questions,
and
we
also
had
to
update
that
like
back-end
golden
path
tutorial
that
I
mentioned
to
deploy
to
kibriya
DS
by
default
and
in
writing.
A
This
documentation
are
kind
of
like
guiding
principles
were
to
assume
as
a
baseline,
that
our
developers
do
not
know
kubernetes,
but
we
did
find
it
like
difficult
to
strike
a
balance,
a
careful
balance
between
informing
them
of
what
kubernetes
is
and
avoid
rewriting
like
the
open
source
documents
out
there
for
kubernetes,
and
so
for
this.
We
basically
leaned
on
linking
to
the
open
source
crew.
Nettie's
documents
were
relevant,
but
we
made
sure
to
like
enhance
it
with
Spotify,
specific
content
and
for
documentation.
A
It's
I
think
it's
pretty
important
to
remember
that
you,
as
the
owners
of
the
offering
or
the
clusters,
are
not
the
intended
audience
for
the
documentation.
Your
understanding
of,
like
kubernetes
or
even
infrastructure
in
general,
does
not
represent
what
your
readers
are
going
to
understand
or
like
approach
your
documentation
with.
A
A
This
is
people
on
the
teams
who
own
the
offering
and
have
designed
it
and
like
work
on
the
infrastructure
behind
it,
basically
being
interrupted
Bowl
during
these
shifts
and
being
responsible
for
answering
questions
and
discussing
with
developers
any
questions
they
may
have
about
the
offering,
and
we
found
that
this
process
can
be
really
mutually
beneficial.
It's
beneficial!
A
A
So
we
built
a
similar
feature
on
kubernetes
via
tugboat,
where
we
would
basically
have
talked
about
deploy
out
a
separate
set
of
resources
like
suffix
with
canary
for
the
resources
like
that
were
appropriate,
but
in
a
way
that
these
replicas
would
still
get
production
traffic,
and
so
the
takeaway
here
is
that
for
your
offering
developers
need
feature
parity
or
better.
So
the
first
part
need
feature
parity,
the
important
operational
things
or
features
that
developers
are
doing
with
the
old
runtime.
A
They
need
to
be
either
able
to
do
the
exact
same
things
or
start
do
the
same
things
or
in
this
case
we're
like
puppet,
not
need
to
do
them
anymore,
and
then,
when
you're
like
deciding
what
features
you
want
to
basically
have
in
place
before
GA,
it
kind
of
depends
on
how
widely
used
the
feature
is
and
like
how
business-critical
it
is.
I
wanted
to
talk
more
about
Canaries
for
this
example,
because
a
lot
of
our
production
services
are
using
Canaries
in
their
workflows
and
a
lot
of
like
business.
A
There's
like
crossing
the
finish
line.
There
actually
is
no
finish
line.
You
know
we're
always
iterating
and
trying
to
improve
things
for
our
offering.
So
the
first
thing
is,
as
I
mentioned,
we
scoped
the
migration
to
stateful
state
lists
back
in
services,
so
we
are
potentially
looking
at
maybe
supporting
staple
services
in
the
future
data
jobs,
consolidation
on
the
data
infrastructure
side
there
are
like
separate
clusters
and
a
separate
offering
for
running
data
jobs.
There's
always
been
talk
of
like
consolidating
the
two
and
then
cron
jobs
as
first-class
citizens.
A
So,
although
we
allow
developers
to
run
cron
jobs
on
their
clusters,
it
definitely
doesn't
have
the
like
first-class
support
that
things
like
deployments
do
because
again
like
it
was
back
in
mic
back
in
stateless
services.
That
was
like
our
primary
scope,
so
this
one's
pretty
near
and
dear
to
me
and
myself
to
myself
basically
supporting
con
jobs
as
first-class
citizens
and
then
higher
level
abstractions.
So,
with
this
right
now,
developers
still
have
to
know
what
a
deployment
is.
What
a
service
is,
what
an
HPA
is
on
some
level.
A
Maybe
one
day
there
will
be
like
a
Spotify
micro
service
custom
resource
that
they
can
use,
and
that
would
abstract
all
of
this
away
from
them,
and
so
I
want
to
cover
some
general
learnings
from
our
migration
and
general
availability
journey
to
the
general
availability
to
end
with.
So
the
first
is
to
be
mindful
and
deliberate
in
dictating
terminology.
A
So
what
I
mean
here
is
that
when
you're
adopting
or
using
kubernetes
you're,
not
really
adopting
or
picking
up
a
single
tool
or
technology,
you're,
really
picking
up
an
entire
ecosystem,
and
with
that
comes
a
lot
of
overloaded
terms
like
deployment
and
services
endpoints,
these
can
get
actually
pretty
confusing
when
you're
discussing
with
developers
or
like
in
things
like
documentation.
So
being
aware
of
this
and
doing
even
what
may
be
considered
little
things
like
always.
A
The
tala
sizing
or
capitalizing
like
resource
types
like
deployment
service
in
your
documentation,
can
go
a
long
way
and
like
alleviating
that
confusion,
migrate
and
stages
or
series
goals
that
increase
in
scope.
So,
as
I
mentioned,
we
definitely
wanted
to
avoid
a
monolithic
migration
for
this
entire
process
and
by
basically
having
like
a
series
of
little
migration
stages
that
had
different
scopes.
A
We
were
able
to
like
discover
and
deal
with
a
lot
of
unknowns
and
like
hidden
problems
at
a
pretty
controllable
pace
like
we
found
out
things
at
this
stage,
found
out
things
at
this
stage
versus
undergoing
a
kind
of
waterfall
monolithic,
migration
and
ending
up
with
a
lot
of
unknowns
hitting
us
at
once,
and
then
the
biggest
learning
for
GA
is
that
developers
will
adopt
and/or
moderate
to
your
kubernetes,
offering
if
it
is
easy,
complete
and
compelling.
So
this
is
kind
of
a
refining
of
the
saying
like.
If
you
build
it,
they
will
come.
A
That's
not
true.
Developers
are
pretty
busy
and
they
hate
complexity
and
pain.
So
if
you
implement
your
offering
in
a
way
that
has
those
things
they're
not
going
to
migrate
to
it,
but
if
you
design
your
user
experience
for
your
kubernetes
offering
in
a
way
that
is
like
easy
to
use,
get
support
for
move
on
to
its
complete
and
the
features
that
developers
are
currently
using
and
perhaps
expect
and
then
also
has,
is
also
compelling
and
that
it
introduces
new
features
that
they
want.
A
For
our
example
like
auto
scaling,
then,
despite
having
pretty
busy
backlogs
developers,
will
find
the
time
to
move
over.
We've
seen
that
firsthand
and
the
last
thing
is
to
talk
to
other
companies
about
how
they
do
infrastructure
and
solve
infrastructure
problems.
It's
one
of
my
another
talk
plug.
But
two
years
ago
one
of
my
colleagues
Dave's
all
the
tusky
give
a
really
great
keynote
on
this.
Basically
about
our
joining
the
CN
CF
and
the
impact
that
has
had
and
basically
like
since
we've
joined
the
CN
CF.
A
We've
greatly
increased
the
amount
of
time
that
we
spend
talking
to
like
our
infrastructure
counterparts
at
different
companies
about
how
they
use
kubernetes,
about
like
infrastructure
problems
in
general,
and
so
in
doing
so,
we've
learned
a
lot
from
them.
They've
learned
a
lot
from
us
and
we've
learned
a
lot
in
terms
of
like
potential
approaches,
pitfalls
to
avoid
that
kind
of
thing,
just
as
I
hope
in
that.
In
sharing
this
talk,
some
of
our
experiences
here
at
Spotify
resonate
with
all
of
y'all.