►
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2023 in Amsterdam, The Netherlands from April 17-21. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Leading Architectural Change with Backstage - Andy Hoffman, Caribou
Yesterday you’re a scrappy startup; today you’re funded and have 12 months to 10x your team and system capacity. The platform, process and architecture doesn’t translate, “the database” is about to fall over, and dozens of new engineers are motivated to innovate in countless different directions, some more stable and secure than others. In this, or any time of change, how do you make sense of the chaos and stimulate the evolution of architecture, processes and platforms without crushing the startup spirit? Backstage, combined with a CNCF stack is a driver of platform scale, stability and governance while still encouraging freedom and innovation. In this session, you’ll hear lessons learned on a DevOps journey to recent unicorn status, with practical strategies for technical and leadership strategies for leading change through Backstage without bureaucracy, and running new services in prod in 5 minutes.
A
C
D
D
So
today
we
will
talk
a
little
bit
about
caribou's
story,
because
I
think
that'll
help
frame
some
of
the
challenges
and
then
we'll
also
talk
a
little
bit
about
how
the
developer
expectations
have
changed
and
how
that
impacts.
How
your
organization
will
change
through
that
we
can
understand
the
obstacles
and
then
start
diving
into
how
you
can
design
a
cloud
native
platform
centering
around
backstage
to
accelerate
the
growth
or
accelerate
the
re-platforming
of
your
organization,
it's
rooted
in
convention
and
it's
rooted
in
iteration.
D
Now,
as
I
had
mentioned
before,
when
I
started
at
Caribou,
we
are
primarily
focusing
on
cash
flow.
How
do
we
plan
a
two
three
year
ramp
up
cycle,
but
the
pandemic
had
some
unintended
consequences
of
people
wanting
to
refinance
their
car
back.
Then
we
were
known
as
Moto
refi,
and
that
was
our
primary
line
of
business.
D
The
application
was
built
on
a
ruby
monolith
and,
to
this
day,
deployments
are
still
painful
with
the
monolith
and
yes,
the
massive
database
still
causes
a
lot
of
n
plus
one
queries.
It's
fun
at
that
time.
Also
we
were
busy.
We
started
moving
off
of
a
SAS
platform
and
a
variety
of
SAS
tools
that
made
sense
at
a
time
when
there
were
only
about
12
engineers,
but
as
we
were
growing,
we
quickly
needed
something
more
robust
from
about
that
time.
D
That
I
started
about
a
week
later
we
got
more
funding,
and
now
we
had
to
accelerate
and
start
rapidly
growing.
We
were
onboarding
engineers
at
a
phenomenal
rate.
It
was
very
confusing
for
a
lot
of
Engineers,
but
we
got
through
it.
We
also
had
10x
growth,
both
in
terms
of
the
business
as
well
as
the
pressure
on
the
system
and,
interestingly,
because
this
was
a
monolith.
D
A
lot
of
the
pressure
came
from
our
internal
employees,
even
though
it
was
also
serving
a
public
website,
and
so
those
sort
of
inefficiencies
and
those
pressures
made
it
so
clear
that
there
was
both
the
the
pressure
cooker
that
caused
us
to
need
to
decompose
as
well
as
this
ability
to
make
it
so
that
we
could
handle
many.
Many
more
Engineers.
D
D
From
the
beginning?
Our
culture
has
been
centered
around
being
metrics
driven
so
that
we
can
also
be
experimental
so
that
we
can
learn
new
things
so
that
we
can
innovate.
Our
CTO
has
been
very
inspired
by
the
Spotify
culture.
We
frequently
reference
some
of
the
open
source,
materials
and
videos,
but
yet
we
have
the
challenge
that
we
are
also
in
a
regulated
industry
because
we're
dealing
with
finances.
We
have
people's
credit
reports.
So
how
do
you
enable
those
things
and
I'm
glad
that
there
were
some
questions
around
that
earlier?
D
Also
the
Cornerstone,
especially
when
you
look
at
a
lot
of
the
door
and
accelerate
research,
is
psychological
safety
when
Engineers
feel
scared,
when
they
feel
that
they
have
to.
You
know,
kill
themselves
basically
to
hit
deadlines
when
they
have
that
pressure
when
they're
afraid
of
making
mistakes.
We
all
know
that's
how
you
have
an
unstable
platform.
D
D
So
how
do
we
then
move
that
change
forward
in
a
slightly
more
organic
way?
And
there
are
certain
challenges
that
keep
that
from
happening.
We've
already
talked
about
the
learning
curve.
A
lot
today,
so
I'm
going
to
kind
of
move
past
that,
but
suffice
it
to
say
I,
think
it
is
impossible
to
guarantee
that
any
engineer
you
hire
has
experience
with
the
complete
Tech
stack
that
you
might
be
working
on.
It's
just
so
diverse.
D
I've
noticed
that
where
we
used
to
be
able
to
say
eh
that
there's
a
little
pain,
it's
a
little
dicey
that
parameter
yeah.
We
didn't
document
that
so
well,
but
you
can
figure
it
out,
because
your
buddy
will
tell
you
that,
because
the
quality
of
the
tools
has
gotten
so
good,
it
is
no
longer
reasonable.
It
is
a
huge
point
of
frustration
from
engineers
when
there
is
that
friction.
D
They
expect
the
same
amount
of
polish
that
they
would
use
on
their
personal
projects
in
that
heavy
regulated
environment,
one
of
the
Lost
opportunities
from
one
of
my
former
places
of
employment
was
when
they
were
starting
to
roll
out
one
of
their
first
Services.
Now
this
was
an
ivr
system
that
the
idea
was
that
you
could
call
in
and
it
would
automatically
send
a
fax
to
your
health
care
provider.
D
So
it
would
not
be
too
difficult
to
pull
out
a
document
service
for
all
of
the
templating
that
needed
to
occur
a
facts,
integration
with
a
third
party
provider
and
deploy
that
as
a
different
container,
make
it
really
easy
to
test
and
monitor
and
then
more
of
a
kind
of
command,
Handler
aggregation,
facts
type
of
service
to
actually
deal
with.
You
know
pulling
all
the
business
logic
together
and
whatnot.
It
seemed
like
it
was
going
to
be
a
slam
dunk,
it
was
not.
D
Instead,
maybe
you
have
seen
this
everything
was
crammed
into
a
single
container.
There
were
not
even
clear
boundaries
between
controllers
and
services,
and
all
this
stuff,
cron
jobs
were
executing
in
the
same
container
that
the
web
server
was
and
business
logic
and
all
the
domain
stuff
was
just
jammed
in
there.
Why
is
this.
A
D
I
disagree
because
of
the
architectural
patterns
that
it
unlocks,
but
people
actually
think
that
kubernetes
is
making
it
unstable
instead
of
more
stable,
which,
of
course,
if
it's
not
configured
right,
that'll
happen.
Also
standing
up.
Services
takes
too
long
because
you
have
to
wire
up
a
whole
bunch
of
different
pipelines.
There's
copy
pasta,
it's
very
confusing
and
it
takes
forever
to
chase
through
those
mounds
of
yaml.
D
But
when
you're
also
trying
to
change
your
organization's
architecture
where
it
used
to
be
clear
what
class
to
go
and
modify
you
now,
don't
know
what
do
I
name
things
do
I
put
it
in
an
existing
service.
Do
I
move
it
somewhere
else
that
simple
question
on
the
surface,
as
we
all
know,
naming
is
one
of
the
hard
problems
and
that
takes
up
time
and
causes
friction
and
for
engineers
who
have
not
had
to
work
in
a
regulated
environment.
D
D
So,
if
you're
going
to
roll
out
a
platform,
you
might
as
well
have
a
little
fun
and
that's
where
we
have
created
antlers
and
any
good
acronym
needs
a
acronym.
So
we
then
came
up
with
automatic
normalized
tool
chain
for
launching
easily
releasable
systems
so
that
we
can
make
it
a
little
fun
and
I'm
glad.
D
Someone
asked
how
much
do
you
need
to
support
this,
because
we
have
about
60
engineers
and
five
devops
engineers
about
nine
twelve
developer
teams,
depending
on
you
know
where
we're
at
and
we're
effectively
filling
three
rolls
of
SRE
of
Automation
and
Dev
tool,
team
and
also
the
kind
of
architectural
advisors
for
a
lot
of
the
teams,
because
we
have
a
lot
of
experience
and
when
I
brought
it
backstage
to
my
team,
they
said.
No,
have
you
lost
your
mind?
D
There
is
no
way
we
have
time
for
this,
so
luckily
we
found
a
managed
provider
with
Roadie
and
Shameless
plug.
They
have
been
fantastic
to
work
with
and
allowed
us
to
be
able
to
actually
adapt
this
quickly
without
having
to
have
the
overhead.
One
of
my
big
fears
actually
was
breaking
apis
and
Patch
maintenance
and
those
sort
of
things
and
the
burden
that
that
would
happen.
D
When
we
start
out,
we
do
not
know
what
the
path
is
kind
of
like
a
mountain
in
the
distance.
You
don't
see
all
the
Craigs
and
valleys
and
the
obstacles
that
will
come
in
your
way,
but
you
at
least
know
which
direction
you're
starting
in
and
our
goal
is
to
make
it
easy
to
know.
Do
I
go
left.
Do
I
go
right
so
that
we
will
wind
up
making
mistakes,
but
they
are
not
the
big
problems,
they're,
the
ones
that
are
regular,
that
we're
all,
of
course,
going
to
make
and
that's
my
cat
Sam.
D
The
key
of
this
is
that,
if
you
have
a
simple
and
intuitive
architecture,
you
can
also
have
a
simple
and
intuitive
implementation
of
scaffolder,
and
so
this
is
where
we
started
rolling
this
out,
because
I'm
trying
to,
in
the
back
of
my
mind,
thinking
through
this
talking
with
my
team
about
this.
How
can
we
avoid
the
challenges
that
we've
seen
before.
D
When
we
decided
to
roll
this
out
and
when
you're
doing
any
architectural
change,
it's
really
important
to
make
sure
that
we
set
boundaries.
What
is
the
scope
of
the
Old
World
versus
the
new
world,
and
how
do
we
onboard
people
because
that
expectation
for
Polish?
If
it's
Rocky,
you
will
lose
confidence.
D
And
so
then,
keeping
it
simple.
We
have
a
management
cluster
that
serves
as
the
control
plane
effectively.
That
then
manages
a
workload
analytics
and
monitoring
so
boundaries
that
we've
decided
make
sense
from
a
you
know,
intrusion
and
risk,
and
all
that
good
security
stuff
and
we're
heavily
relying
on
Argo,
CD,
Argo,
workflows
and
infrastructure
as
code
that
with
Argo
CD,
pretty
much.
Everything
in
our
environment
is
modeled
as
an
application,
including
workflows
and
including
terraform
jobs
and
whatnot.
Everything
is
yaml
and
templatable.
D
Now
we
could
have
done
something
more
complex,
but
the
reality
is.
You
have
to
gauge
the
speed
that
you
need
to
move
things
out
with
and
then
the
learning
curve
of
your
own
team
for
implementing
more
complex
models
and
great
ammo's
easy
enough,
and
so
then,
through
that
Argo
can
deploy
any
of
our
resources
and
because
we
also
take
advantage
of
config
connector
or
we
could
have
just
as
easily
used
cross
plane.
D
The
key
here
is
when
a
yaml
engineer
or
devops
engineer
doesn't
think
through
convention.
You
will
be
entering
the
World
of
Pain
and
I.
Think
we've
all
been
there,
and
so
that's
where,
if
you
can
build
intuition,
if
you
can
build
your
architecture
into
the
structure
of
everything
so
that
it
permeates
your
entire
environment.
D
D
From
a
standpoint
of
how
do
you
organize
that
architectural
change
PR
workflows-
and
this
is
fantastic,
so
basically,
you
can
have
abstractions
for
your
engineers
to
Think
Through
of
as
domains
and
then
set
rules.
What
are
the
things
that
should
be
shared
within
a
domain?
What
are
the
things
that
should
be
shared
within
a
given
system?
What
are
the
things
that
should
be
shared
in
a
component?
D
Hopefully
not
too
much,
but
basically
we
can
say
that
a
PR
opens
a
or
a
a
template
simply
opens
a
PR
and
a
parent
repository,
and
then,
through
that
workflow
you
can
have
governance.
You
can
have
that
ability
to
say
the
staff
Engineers
sign
off
on
New
domains,
but
you
know
the
teams
can
create
new
systems,
and
certainly
any
engineer
can
create
new
components,
creates
those
clear
boundaries
and
it
creates
an
idea.
You
know
a
hook
to
hang
your
projection
of
what
that
system
on,
so
to
speak.
D
I
mentioned
the
pr
workflow,
but
then
also
because
we're
using
Argo,
workflows
and
Argo
events
to
capture
all
of
our
GitHub
events.
We
can
have
any
number
of
checks
running
all
baked
into
our
templates.
We
can
have
audit
Trails
coming
off
the
changes
from
Argo
and
from
GitHub
and
all
of
this
information,
as
well
as
I'm,
very
excited.
Now
we
get
to
lock
down
our
GitHub
repositories
and
have
a
full
audit
Trail
being
able
to
trace
the
identity
of
who
creates
the
different
template.
Things
I'm
very
excited
about
that
feature.
D
So
then
we
have
this
full
picture
of
what's
happening
in
our
system
and
because
it's
all
baked
in
to
get
and
baked
into
really
easy
to
track
data
we've
solved
a
lot
of
these
problems
and
in
the
future,
hopefully
q1
or
Q2.
We
start
rolling
out
Opa,
because
then
we
can
also
validate
and
have
some
emissions
controllers
to
get
that
extra
layer
of
concern
for
making
sure
that
our
resources
are
what
we
would
expect,
because
we
were
trying
to
move
fast.
We
had
to
choose.
D
I
would
have
loved
to
have
been
able
to
spend
more
time
rolling
out
custom
operators
and
resource
definitions,
but
the
reality
is,
we
didn't
have
time
for
that
and
I
think
a
lot
of
us
get
in
the
position
where
we
don't
have
time.
So
what
is
good
enough,
and
can
you
make
small
modular
components
that
have
those
best
practices?
It
can
be
from
your
database
to
your
cloud
storage
to
your
deployments.
D
D
If
you
need
command
query
cool,
you
can
do
that
and
provide
those
basics
for
your
team
pretty
easily
and
because
you
follow
semantic.
Versioning
I
would
hope
that
now,
if
you
treat
this
as
each
template
maps
to
an
umbrella
Helm
chart,
now
you
can
add
in
your
own
customizations
into
that
Helm
chart.
And
then
all
you're
really
having
to
deal
with
to
template
out
is
some
of
the
parameters
and
the
values
file.
D
It's
too
big
of
learning
curve.
It's
one
click
you
can
customize
it,
but
it's
one
click
architecture
is
unintuitive.
Well,
it
becomes
more
intuitive
when
you're
following
conventions
there's
too
much
to
do
well,
you
have
fewer
decisions
and
the
unfamiliar
patterns
are
solved
by
codifying
those
best
practices
in
that
code.
D
So
now
we're
spending
our
time
not
doing
operational
stuff,
that's
low
value
and
spending
our
time
doing
higher
value
stuff,
making
more
templates,
documentation,
training,
leveling
up
engineers
and
creating
a
learning
culture
and
pro
tip
and
I.
Think
some
of
the
talks
later
this
afternoon
might
have
some
interesting
things
with
this.
D
D
The
whole
idea
is
that
we
iterate
and
we
mature,
that
we
work
on
building
a
ton
of
different
modular
components
and
we've
recently
seen
other
teams
start
adopting
this.
So
our
data
pipelines
previously
were
all
this.
You
know
big
batch
ETL
and
the
data
engineering
team
has
taken
it
amongst
themselves
to
build
event-driven
templates
so
that
now
you
can
simply
add
on
sidecars.
D
So
as
we
deprecate
old
domain
nomenclature
and
try
to
move
to
New,
that's
all
tracked
and
we're
working
on
making
that
visible
through
the
tech
radar.
That's
something
that
we're
actively
working
on
implementing
now,
because
the
goal
is
to
manage
expectations
and
make
sure
that
we
are
aware
of
communicating
pain
points
and
where,
in
the
life
cycle,
these
things
are
so
that
the
engineers
will
understand
if
they
will
run
into
hiccups
if
they
will
need
to
budget
extra
time
for
these
things,
so
they're
more
predictable
and
have
a
far
more
positive
experience.
D
And
finally,
what
I
would
like
to
close
on
is
thinking
about
the
real
reason.
Change
is
hard
and
Buddhist
philosophy.
There
are
three
different
categories
of
change,
one
of
which
is
the
suffering
of
change.
I'm,
sorry,
three
different
categories
of
suffering,
one
of
which
is
the
suffering
of
change.
D
B
B
The
quality
of
of
catalog
entities,
and
so
what
I
mean
by
that
is
there's
like
the
bare
minimum
that
you
need
to
register
something
in
backstage.
But
then
you
may
decide
that
certain
plug-in.
You
know
that
the
the
entities
aren't
as
valuable
as
they
could
be
without
certain
plugins
implemented
like
Tech.
Docs
is
a
good
example
of
that
yeah.
C
B
D
So
when
we
create
a
new
template,
we
also
have
all
the
properties
that
we
would
expect
for
minimal
Acceptance
in
there
and
what
we
are
increasingly
doing
is
highlighting
how
teams
who
are
using
these
tools
are
having
more
Effectiveness
and
less
pain
points.
And
so
then
there
is
some
level
of
the
oh
cool
I.
Want
this
feature.
Oh,
what
did
you
need
to
do
to
do
that?
D
C
D
It
pretty
much
prevents
it,
because
the
reason
why
tiger-like
architectures
wind
up
coming
about
is
generally
because
teams
think
that
the
barrier
of
creating
a
cat-like
architecture
is
too
hard,
at
least
in
my
experience.
The
other
thing
would
be
the
knowledge
gap
of
borking
up
with
the
boundaries
of
a
context
and
those
sort
of
things,
and
that's
where
we've
tried
to
create
an
intuition
so
that
if
by
pre-loading
a
whole
bunch
of
domains,
this
is
kind
of
work
in
progress.
D
Actually
we're
hoping
that
that's
going
to
be
able
to
create
some
boundaries
so
that
it's
more
in
intuitive
to
not
accidentally,
say
I'm,
just
going
to
go
ahead
and
cram
something
in
another
context
that
is
going
to
wind
up
creating
architectural
complexities
or
challenges,
because
a
lot
of
that
starts
to
starts
to
come
up
when
good
intentions
created
a
context.
But
then
you
really
need
to
start
thinking
about
a
new
domain
and
or
starting
thinking
about
the
type
of
sequence.
Diagrams
actually
I
mean
a
lot
of.