►
From YouTube: Shifting Spotify Engineering from Spreadsheets to Backstage - Johan Haals & Patrik Oldsberg, Spotify
Description
Don’t miss out! Join us at our next event: KubeCon + CloudNativeCon Europe 2022 in Valencia, Spain from May 17-20. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Shifting Spotify Engineering from Spreadsheets to Backstage - Johan Haals & Patrik Oldsberg, Spotify
Is your bookmark bar filled to the brim with links to internal infrastructure tools? Are you using spreadsheets to track the state of your software and infrastructure? Does your company suffer from fragmentation, like hundreds of startups glued together? Then this talk is for you! It will tell the story of how Spotify created a service catalog to bring order to thousands of microservices. It grew to become Backstage, an internal developer portal supporting not just services, but also machine learning models, documentation, over 9000 data pipelines, and much more. With 115 plugins contributed by 58 different teams, Backstage is now Spotify’s single pane of glass for the entire engineering organization. In 2020 Backstage and its software catalog was open sourced and donated to the CNCF, making it available for everyone to use. The talk will also walk you through strategies for adopting Backstage in an existing organization, and how to bootstrap your own catalog.
A
Lastly,
we'll
share
our
learnings
and
help
you
get
started
with
backstage,
but
before
we
start
here
is
a
brief
introduction
to
us.
My
name
is
johan
and
I'll
be
co-presenting
this
session,
together
with
patrick
we're,
both
engineers
at
spotify
and
have
spent
many
years
building
infrastructure
platforms.
We
currently
work
as
open
source
maintainers
of
the
backstage
project.
A
A
A
A
And
as
years
passed,
we
ventured
into
this
century
where
microservices
were
invented.
Our
engineering
team
had
grown
into
the
size
of
a
small
village
and
we've
even
convinced
some
of
our
platform
engineers
to
learn.
Html
the
stone
age
monoliths
were
starting
to
break
down
into
microservices,
but
with
this
shift
or
engineers
starting
having
trouble
to
keep
track
of
all
the
pebbles
and
since
server
db
only
knew
about
hardware,
we
had
no
place
to
keep
track
of
all
the
new
services.
A
So
we
built
service
db,
which
in
true
microservice
fashion,
was
completely
separate
from
serverdb,
and
here
you
can
see
servicedb
in
action
with
its
beautiful
bootstrap
ui.
It
allowed
engineers
to
browse
services
as
well
as
figure
out
where
they
were
deployed.
Of
course,
it
was
written
in
python
and
in
coffeescript,
which
was
the
hotness
of
the
time.
B
Our
platform
organization
also
started
inventing
a
whole
new
set
of
tools
for
our
engineers
to
enjoy.
They
were,
however,
scattered
across
a
multitude
of
different
sites,
and
many
of
them
were
hard
to
find
without
the
tribal
knowledge
of
where
they
were.
We
also
wanted
to
know
more
about
the
service
than
simply
where
it
was
deployed.
Things
like.
Is
it
running
in
production
who
owns
this
service
and
who's
on
call
for
it
at
the
moment?
B
So
as
a
solution
to
all
of
these
projects,
we
built
system
c
staying
true
to
fashion.
At
the
time
it
was
written
in
angular,
rather
than
coffeescript
spotify
engineers
now
had
this
place
where
they
could
browse
through
all
the
services,
but
when
you
logged
in
you
were
presented
with
a
list
of
all
of
the
services
that
you
owned,
and
you
also
had
the
possibility
to
create
new
services.
B
Each
service
in
the
catalog
also
had
its
own
dedicated
space.
There
was
an
overview
page
with
general
information
like
metadata
things
to
things
like
links
to
external
sites
and
contact
and
paging
information
for
each
service.
There
were
also
additional
tools
like
build
history,
capacity,
management,
deployment,
status
and
traffic
routing.
B
B
B
We
again
started
seeing
fragmentation
as
developer
tools
started
being
built
outside
of
system
c,
especially
in
other
domains
like
data
engineering
and
machine
learning
that
really
hadn't
been
brought
in,
but
we
didn't
give
up.
We
doubled
down
on
consolidating
all
of
our
engineering
tools
in
one
place,
with
a
focus
on
reducing
fragmentation
and
speeding
up
onboarding
of
our
new
engineers
to
alleviate
our
current
struggles
system
c
was
reworked
to
make
it
easier
to
scale
development
across
multiple
teams.
B
B
B
One
of
the
metrics
we
use
to
measure
productivity
at
spotify
is
onboarding
time
of
our
new
engineers.
More
specifically,
the
time
it
takes
for
an
engineer
to
send
their
10th
pull
request.
This
method
metric
had
been
increasing
steadily.
As
our
organization
grew,
developer,
tooling,
had
become
more
fragmented
and
it
got
harder
to
find
information.
B
B
You
can
see
that
we
bring
delivery
status
and
access
control
to
the
foreground,
while
for
a
service
we
might
have
highlighted
things
like
the
deployment
status.
There's
also
a
whole
different
set
of
tools
available
through
the
top
bar
and
thanks
to
the
plugin
system.
This
page
and
all
of
the
tooling
could
be
iterated
on
in
isolation
by
our
data
platform
teams.
B
A
benefit
of
using
one
interface
for
all
of
this
tooling
is
that
it
makes
it
much
easier
for
engineers
to
venture
into
other
domains
to
broaden
their
knowledge.
For
example,
a
backend
engineer
can
now
easily
manage
data
while
building
a
recommendation
engine
staying
in
the
same
familiar
interface
that
they
use
for
their
services.
B
A
A
The
way
you
adopt
backstage
is
perhaps
different
from
other
cncf
projects.
It's
not
an
off-the-shelf
docker
container
that
you
deploy
straight
to
your
kubernetes
cluster.
Instead,
we
supply
tooling,
to
help
you
create
manage
your
own
project.
It's
an
application
framework
that
you
extend
with
plugins,
both
ones
built
by
the
community
as
well
as
internal
ones.
A
A
So,
let's
imagine
that
I'm
an
engineer
interested
in
a
particular
service
called
the
greeter.
If
I
had
to
backstage
and
look
it
up,
I'm
greeted
with
this
overview
page
here
I
can
see
a
collection
of
different
cards,
giving
me
general
information
about
the
service
showing
things
like
latest
github.
Workflow
runs
links
to
documentation
and
on-call
information
from
pagerduty.
A
Each
of
these
cards
is
provided
by
its
own
plugin
and
you
can
access
more
at
the
top
bar.
Basically,
this
page
take
information
and
tooling
scattered
across
your
bookmark
bar
and
makes
it
available.
In
one
view,
the
plugins
that
you
see
on
this
page
are
selected
based
on
the
type
of
component,
we're
looking
at
in
this
case
a
service,
and
you
might
have
different
set
of
toolings
for
things
like
websites
or
libraries.
A
A
We
call
each
instance
of
these
things.
An
entity
a
component
is
a
kind
of
entity
defined
by
the
kind
field
in
the
yaml
and
so
far
we've
talked
a
lot
about
components,
but
backstage
aims
to
mold
the
bigger
picture,
so
components
can
be
part
of
a
system,
depend
on
resources
and,
of
course,
implement
apis.
A
It's
important
to
mention
that
this
is
not
a
model
that
is
set
in
stone
for
you
as
an
adopter
of
backstage.
It
can
be
reworked
or
extended
to
fit
your
organization
just
like
custom
resource
definitions
in
kubernetes
in
our
own
installation,
for
example,
we
model
things
like
native
applications
and
machine
learning
models
so
going
back
to
the
yaml.
A
A
So,
let's
look
at
how
this
data
is
used
to
build
out
the
front
end.
The
highlighted
sections
in
this
example
drive
different
parts
of
the
ui.
The
top
links
section
produces
a
small
link
card
on
the
overview
page
and
the
page
duty
annotations
causes
the
page
of
duty
card
to
be
shown.
The
pagerduty
plugin
then
uses
the
value
of
the
annotation
in
order
to
associate
an
entity
with
a
service
in
pagerduty.
A
This
pattern
is
something
we
frequently
use
where
we
keep
a
lightweight
reference
between
an
entity
and
an
external
system.
Plugins
then
use
that
reference
to
fetch
more
data
or
take
actions.
In
general,
we
strive
to
keep
integrations
lightweight,
storing
only
what's
necessary
in
the
catalog
and
something
we
haven't
talked
about.
Yet
is
how
engineers
add
entities
to
the
catalog,
and
one
way
is
to
use
manual
registration
of
a
yaml
file
in
the
ui
the
locations
get
stored
in
the
catalog
and
are
then
used
to
ingest
the
metadata.
A
They
help
you
automate
the
creation
of
new
projects
along
with
taking
care
of
additional
tasks
such
as
registration
at
spotify.
We
use
software
templates
to
save
engineering
time
and
drive
standardization
they're
a
great
way
to
encode
common
patterns,
while
also
helping
engineers,
you
run
a
template
by
filling
out
a
form
that
it
generates,
which
in
turn
executes
a
series
of
actions.
A
These
actions
can
range
from
creating
a
new
project
in
your
version
control
system
to
manage
more
specialized
tasks
like
creating
a
monitoring
bundle
in
grafana.
It's
a
bit
similar
to
github
actions.
If
you
ever
use
that
and
we
won't
dive
further
into
github
or
into
software
templates
in
this
talk
as
they
could
really
have
a
talk
of
their
own.
B
It's
important
to
note
that
the
catalog
itself
is
not
the
source
of
truth
here.
It
serves
as
a
collector
and
cache
of
information.
However,
processors
can
do
much
more
than
just
reading
yaml.
They
can,
for
example,
handle
validation
or
call
out
to
external
systems
to
decorate
or
even
generate
new
entities.
The
catalog
does
give
you
a
default
set
of
processors,
but
writing
your
own
ones
is
a
key
point
of
customization
when
adopting
backstage.
B
B
B
B
You
can,
for
example,
point
your
catalog
at
the
github
organization
and
have
it
ingest
all
of
the
catalog
infoyaml
files
that
it
can
find.
This
is
great
because
it
skips
the
manual
step
and
it
makes
your
catalog
a
bit
more
resilient
as
you're
not
relying
to
the
date,
relying
on
the
database
to
store
their
locations.
B
Another
convenient
convenient
method
is
through
configuration
when
deploying
the
catalog.
You
can
provide
it
with
a
list
of
locations
for
it
to
include
this
method
is
great.
If
you're
the
one
operating
a
backstage
instance-
and
you
can,
for
example,
use
this
to
lock
down
who
is
allowed
to
add
entities
of
a
certain
kind
through
rules
in
the
catalog.
A
typical
use
case
for
this
is
to
statically
configure
all
of
your
software
templates
and
then
forbid
usings
for
users
from
registering
any
more
of
those.
B
B
Let's
look
at
the
more
concrete
example
of
automation
using
a
catalog
processor
tech
docs
is
another
core
feature
of
backstage
and
it
takes
markdown
documentation
that
lives
with
your
code
and
makes
it
searchable
and
readable
within
backstage
to
enable
tech
talks
for
your
component.
You
need
to
add
a
specific
annotation
to
your
catalog
info
file,
which
in
turn
then
makes
the
documentation
tab
show
up
now
within
an
organization.
B
B
This
example
also
highlights
how,
when
you
read
entities
back
from
the
catalog,
they
will
contain
more
information
than
your
source,
yaml
files.
This
is
similar
to
how
kubernetes
works,
where,
when
you're
reading
an
object
back
it
there,
it
has
more
information
than
the
original
manifest
that
you
applied.
B
Now,
let's
have
a
look
at
the
demo
of
backstage,
so
I
have
my
own
local
backstage
instance.
We
have
chosen
to
use
the
catalog
index
page
as
our
home
page,
and
this
is
where
we
can
browse
all
of
the
components
in
our
catalog
right
now,
I'm
looking
at
all
of
the
components
that
I
own,
but
I
can
also
switch
the
filter
over
to
see
all
of
the
entities
in
the
in
the
organization.
B
B
B
B
B
But
let's
head
back
to
our
greater
component
now
we
talked
about
a
lot
of
open
source
plugins
that
you
can
install
in
in
backstage.
So
let's
try
installing
one
of
those
we'll
have
a
look
at
the
to
do
plugin,
which
is
able
to
scan
your
source
code
for
to-do
comments
and
then
make
them
browsable
within
backstage.
B
B
B
B
B
I,
as
a
selfish
engineer,
should
save
time
by
adding
my
service
to
the
catalogue,
use
carrots
rather
than
sticks.
As
an
example,
at
spotify,
a
newly
created
service
in
backstage
gets
monitoring,
logging,
cicd,
cost
insights
and
much
more
out
of
the
box.
It's
important
that
being
part
of
the
catalog
does
not
become
a
chore.
B
Speaking
of
bringing
together
all
development
development
tooling,
the
next
strategy
is
to
start
small.
You
might
be
tempted
to
build
out
a
bunch
of
new
fancy
plugins
for
your
existing
internal
tools,
but
keep
it
simple.
Often
what
you
want
to
start
with
is
just
a
link
out
to
your
existing
tool.
Then,
over
time,
as
you
build
out
your
backstage
instance,
you
can
start
bringing
information
and
actions
from
these
tools
into
backstage
and
eventually
completely
replace.
The
tools
also
be
sure
to
check
out
all
our
open
source
plugins
that
have
been
built
by
the
community.
B
A
A
They
contributed
to
all
parts
of
the
project
and
helped
build
out
backstage
to
what
it
is
today
and
it's
been
growing
more
than
we
could
ever
imagine
also
be
sure
to
check
out
our
discord
server
where
all
the
maintainers
and
community
members
hang
out.
You
can
find
links
to
all
of
that
on
the
website
and
thank
you
so
much
for
listening
and
enjoy
the
rest
of
your
day.