►
Description
Presented by Rob Zuber, CircleCI.
As companies scale, they’re likely to break things out into microservices. When that time comes, engineers are too eager to embrace their newfound freedom, throwing out the good with the bad. But this clean slate approach can have a severe impact on team velocity. Conventional wisdom maintains that decoupling reigns supreme, but the truth is that there’s a spectrum, depending on team size and complexity of the codebase. Learn how CircleCI walks the line between monolith and microservice, as CTO Rob Zuber discusses how we optimized service creation by embracing consistency and shared tooling.
About GitHub Universe:
GitHub Universe is a two-day conference dedicated to the creativity and curiosity of the largest software community in the world. Sessions cover topics from team culture to open source software across industries and technologies.
For more information on GitHub Universe, check the website:
https://githubuniverse.com
A
All
right,
we
ready
I,
want
to
talk
to
you
today,
a
little
bit
about
microservices
and
we
at
circle.
Ci
are
big
believers
in
the
blameless
postmortem.
If
you're
not
familiar
with
blameless
post-mortems,
the
notion
is
effectively
that
you
want
to
get
everything
out
on
the
table,
and
so
you
take
the
people
out
of
the
equation
and
just
discuss
what
happened
the
process
and
things
that
you
could
have
done
better
and
use
that
as
a
great
opportunity
for
learning.
A
So
in
this
subject
today,
I
want
to
focus
on
what
we
learned
as
we
went
through
a
transition
from
a
monolith
into
micro-services
and
some
of
the
places
that
we,
when
we
look
back,
could
have
done
better
to
help
you,
depending
on
where
you
are
in
the
cycle
either
heading
into
this
transition
or
even
if
you're,
deep
into
using
microservices,
understand
some
places
that
you
could
make
some.
Hopefully
some
improvements
to
how
you
work
in
order
to
do
this,
I
need
to
take
a
little
bit
of
a
step
back
and
just
help.
A
You
understand
Circle,
C,
I
and,
and
how
we
got
to
here
so
Circle
CI
is
a
continuous
integration
and
continuous
deployment
platform.
We
offer
our
service
both
in
a
cloud
environment
and
hosted
on
your
own
servers
in
our
cloud
environment.
We
run
about
I,
think
we
just
broke.
Five
million
builds
a
month,
so
it's
a
pretty
large
platform
and
we
certainly
didn't
come
out
of
the
gate
as
a
business
prepared
to
take
on
this
kind
of
load.
A
We've
had
to
make
a
number
of
changes
in
our
infrastructure
and
architecture
over
the
years,
we're
about
six
years
old
at
this
point,
a
little
over
six
years
old,
and
so
when,
at
the
time
that
I
joined,
which
was
the
middle
of
2014
Circle
CI,
was
built
entirely
as
a
monolith.
It
was
the
the
right
solution
to
the
problem.
At
the
time
we
were
still
figuring
out
our
business
and
we
basically
used
that
single
piece
of
code.
A
This
is
the
definition
of
a
monolith
to
do
everything
that
we
did
to
respond
to
web
requests
to
process
jobs
to
execute,
builds
to
send
notifications
out
about
those
builds
when
they
were
complete,
and
this
was
great-
we
were
able
to
move
relatively
quickly
and
and
get
things
done,
but
as
we
progressed,
we
start
to
hit
really
the
limits
of
that
that
architecture,
and
so
this
is
not
actually
a
chart
from
data
dog.
It's.
But
it's
imprinted
on
my
brain,
because
I
saw
this
pretty
regularly,
and
so
what
happened
was
as
we
grew.
A
We
got
to
a
point
where,
when
we
added
additional
capacity
in
the
form
of
hosts,
when
we
added
more
systems
into
our
platform,
we
were
actually
able
to
run
fewer
builds.
So
this
is
what
it
looked
like.
As
we
scaled
up,
we
were
able
to
process
fewer
and
fewer
builds
at
any
given
time
because
of
this
monolithic
architecture.
A
So
basically,
the
each
system
was
thrashing
on
the
queue
trying
to
take
the
next
job
that
another
system
would
take,
and
so
a
bunch
of
time
was
wasted
and
we
had
to
start
thinking
about
making
some
changes
and
and
soon
after
that,
in
July
of
2015,
we
had
a
pretty
massive
outage.
It
was
24
hours,
long
I
wouldn't
wish
that
on
anyone,
and
we
started
to
realize
a
lot
of
the
technical
and
architectural
debt
that
we
really
needed
to
start
addressing
in
our
platform.
A
So
we
we
started
out
incrementally
we
made
some
small
changes
to
buy
ourselves,
some
breathing
room,
some
Headroom,
but
as
we
did
that
we
realized,
we
were
headed
on
a
path
that
was
just
incrementing
towards
a
local
maximum
and
we
needed
to
make
some
big
big
changes,
but
as
in
particular
in
our
build
engine,
how
we
scheduled
jobs
and
how
we
executed
them.
And
so
we
had
to
take
some
time
and
think
about
how
we
could
do
that
in
a
way
that
wasn't
going
to
impact
all
of
our
customers
as
we
started
to
do
it.
A
So
we
wanted
to
create
this
seam
effectively
in
and
split
jobs
between
what
we
now
call
our
1.0
platform.
No
one
thinks
of
their
first
platform
as
one
auto
and
what
we
eventually
called
2.0,
and
so,
as
we
started,
we
could
run
one
build.
Two
builds
three
builds
through
this
other
build
infrastructure
and,
of
course,
those
logical
blocks
of
API
wander
result
processing,
that's
actually
just
our
existing
monolith.
A
A
The
kind
of
engineers
were,
the
best
thing
that
you
can
do
is
just
get
out
of
their
way,
so
we
knew
we
could
trust
these
engineers
to
figure
out
not
just
what
our
build
engine
should
be
and
how
it
should
work,
but
also
how
we
should
be
going
about
building
software
in
the
future
and
and
they
did
a
great
job.
The
results
were
amazing.
We
had,
you
know,
really
big
impact
on
the
business.
A
We
were
able
to
move
quickly
and,
and
they
were
able
to
really
demonstrate
some
of
the
value
that
we
could
get
out
of
moving
into
micro
services.
At
the
same
time,
our
monolith
was
really
starting
to
creak,
just
not
only
from
an
operational
perspective
but
from
an
engineering
perspective.
So,
as
developers
were
working
on
that
monolith,
the
projects
were
taking
longer
and
longer.
A
New
engineers
would
come
in
and
just
the
work
of
learning
the
full
context
of
the
changes
that
you
were
going
to
make
as
it
was
spread
throughout
the
monolithic
codebase
was
becoming
a
big
problem.
So
seeing
the
difference
between
how
things
were
working
in
the
micro
services
world
versus
what
we
had
achieved
or
what
we
were
achieving
in
the
monolith
in
the
monolith
world,
we
said.
Okay,
we
should
probably
be
thinking
about
doing
more
things
as
micro
services,
but
we
wanted
to
be
cautious.
A
We'd
heard
a
lot
of
tales
of
poor
decision-making
heading
into
micro
services
and
I
had
experiences
myself
in
a
previous
company
where
we
didn't
truly
understand
the
business.
So
we
created
micro
services
to
serve
specific
pieces
of
our
architecture
that
ended
up
being
the
wrong
pieces
and
when
you
have
the
wrong
pieces
of
your
architecture
split
out
into
micro
services,
you
just
create
a
lot
of
overhead
for
yourself.
A
Additionally,
we
truly
understood
the
operational
characteristics
of
our
platform.
At
this
point,
I
mean
we're
four
to
five
years
into
our
lifecycle,
as
a
company
there's
very
low
risk
that
we're
gonna
pivot
into
another
business.
We
just
are
trying
to
scale
this
thing
operationally
and
we're
scaling
our
engineering
teams,
so
we're
good
to
go
right,
let's
jump
in,
and
let's
do
this
micro
services
thing,
we've
checked
the
boxes.
So
what
happens
immediately?
A
The
wheels
come
off
the
wagon
like
it
is
absolute
grinding
to
a
halt
in
terms
of
our
engineering
productivity,
and
it's
worth
noting
at
this
point
that
we
are
a
CI
and
CD
company.
We
spend
every
single
day
thinking
about
developer
productivity,
not
just
because
we
want
our
own
engineering
team
to
be
productive,
but
because
that
is
the
service
that
we
provide
to
all
of
our
customers.
So
this
isn't
just
a
question
of
developer
productivity
and
and
my
own
team.
A
A
We
should
just
build
this
next
feature
in
the
monolith,
but
we
knew
all
of
the
problems
we
were
having
in
the
model
list,
so
we
really
wanted
to
sit
down
and
figure
out
where
things
had
gone
wrong
and
one
thing
worth
this
is
my
favorite
architecture
diagram
ever,
but
one
thing
that
that
really
came
out
in
this
process.
This
is
surely
not
about
identifying
what
those
services
are,
but
it's
the
shaded
region.
When
you
build
in
a
monolith,
you
come
in
to
work
every
day
and
work
on
business
logic.
A
You
build
features
for
your
customers,
because
the
work
of
connecting
to
a
database
and
exposing
an
HTTP
stack
and
retrying
bad
calls
over
the
network.
Somebody
wrote
that
before
you
even
showed
up
at
this
company,
you
don't
think
about
that
stuff.
Ever
you
just
show
up
and
work
on
business
logic
every
day
and
you're
improving
the
functionality
for
your
customers
move
over
here
to
the
to
the
micro
services
world
and
honestly,
those
shaded
regions
could
probably
be
75%
of
those
services.
A
A
So
you've
gone
from
there's
a
small-town
comfort
of
working
in
the
monolith
and
you've
grown
up
and
moved
to
the
big
city,
and
now
there's
this
unfamiliar
surrounding
and
you're
trying
to
understand
even
how
to
make
things
work
day
to
day,
and
so
the
amount
of
time
that
you
can
spend
on
that
business
logic
is
greatly
greatly
reduced.
So
when
we
sat
and
thought
about
what
we
had
done
in
throwing
people
into
this
new
world,
we
identified
three
key
sources
of
friction,
and
that
was
decision-making,
novelty
and
repetition.
A
Let
me
talk
to
you
first
about
decision-making
oops,
so
everyone
is
familiar
with
analysis,
paralysis,
I
think
in
the
notion
that
you
could
be
faced
with
the
decision.
That's
so
complex
that
you
spend
all
this
time
and
never
really
get
to
a
decision
either,
because
there's
too
many
choices
or
because
the
choices
are
too
complicated,
honestly,
but
at
a
higher
level,
there's
just
some
overhead
to
needing
to
make
a
decision
and
I
think
this
is
a
great
example
of
where
we
do
this
in
software
development.
A
So
in
my
third
or
fourth
year
of
my
software
development,
career
I
started
working
on
a
really
large
C
project.
It
was
well
larger
than
this
for
sure,
and
this
was
the
code
style
that
I
showed
up
and
saw,
which
is
which
was
super
weird
to
me
and
the
great
thing
about
being
in
your
third
or
fourth
year
of
your
software
development
career.
A
I
need
to
go,
find
the
lead
engineer,
explain
to
him
that
this
is
wrong
and
we
can
reformat
the
codebase,
because
that's
that's
high
value,
but
good
news
for
me
was
that
I
was
actually
super
intimidated
by
this
guy
I
think
I
still
would
be
if
I
ran
into
him
again
and
I
totally
revered
him.
So
I
thought
maybe
that
he's
doing
something
right
and
I
just
don't
understand
it.
A
So
why
don't
I
just
see
if
I
can
get
my
editor
to
indent
brackets
in
this
way
and
of
course,
I
could
because
well
I'm,
not
gonna,
tell
you
what
editor
I
use,
but
it
was
really
easy
and
all
of
a
sudden
the
concept
of
how
I
should
format
my
code
disappears
into
the
background
and
I
start
thinking
about
the
code
that
I'm
working
on
and
that
to
me
is
a
very
good
example
that
we
use
all
the
time.
I
mean
I'm,
sure
all
assume
many
of
you
out.
A
There
have
coding
standards
for
your
code
bases
so
that
you
can
focus
on
the
things
that
really
really
matter.
So
that's
great
in
the
small.
What
about
moving
into
micro
services?
Why
didn't
we
think
that
this
was
going
to
be
a
problem?
Well,
we
made
a
bunch
of
decisions.
We
made
some
decisions
early,
so
we
are
a
closure
shop,
and
so
we
said
it
is
not
up
in
the
air
for
you
to
decide
in
your
service
what
language
our
stack
you're,
going
to
use
your
going
to
use
closure.
A
We
everyone
here
knows
it
it's
treated
as
well:
let's
not
waste
time
on
that.
We
used
G
RPC
as
a
communication
protocol
in
our
first
services
that
we
built
in
this
Trudeau
project,
it
was
working
really
well,
we
said
consistency
is
great.
Let's
use
G
RPC
we're
going
to
use
Postgres
as
data
storage,
we're
going
to
use
docker
on
top
of
kubernetes
for
deploying
our
applications.
So
that's
a
lot
of
important
decisions
that
we
made
coming
out
of
the
gate
and
we
felt
like
we
put
real
constraints
on
this
process.
A
Well,
there
are
a
lot
of
decisions
to
be
made
in
software
development
it
turns
out
and
you
can
make
them
very
very
differently.
So
one
example:
we
used
to
use
a
library
called
schema
which,
if
you're,
not
a
closer
developer,
you
might
not
know,
but
it
is
effectively
a
typing
system
ish
for
enforcement,
of
the
shape
of
your
data
structures
since
that
had
come
out
and
since
Circle
C
I
had
used
it
a
similar
set
of
patterns
and
tools
had
been
introduced
into
the
closure
core
language,
and
so
one
of
our
teams
said.
A
Oh,
this
looks
interesting.
Maybe
we
should
use
that
now
I'm
faced
with
a
decision.
Not
only
is
it
a
decision
that
I
have
to
make,
that
would
be
great
to
just
wipe
off
the
table
because
it
doesn't
really
matter,
you
could
use
either
and
you'd
be
fine,
but
suddenly
you
realize
oh
I
need
closure,
1.8,
not
closure
1.7,
which
we've
been
using,
so
let's
switch,
but
we
have
a
bunch
of
libraries
that
have
only
ever
been
tested
on
1.7.
A
So,
let's
spike
out
using
this
entire
new
change
to
our
core
infrastructure,
because
we
want
to
use
a
different
library
and
the
value-add
of
all
of
that
work
again
compared
to
shipping
features
to
your
customers
is
very,
very
low.
So
the
key
point
that
we
took
away
from
this
was
reduce
your
decision-making
to
exceptions,
be
very
clear
and-
and
one
thing
I
would
highlight
here-
is
you
can't
come
completely
top-down
with
these
kinds
of
decisions?
A
Because
when
you
do
that,
people
are
just
going
to
disregard
your
decisions
anyway,
so
we
actually
got
together
a
group
and
first
I
walk
them
through
Jeff
Bezos
is
disagree
and
commit
pattern,
evil
or
whatever
you
want
to
call
it
mantra,
which
was
basically
we
as
a
group
are
going
to
agree
to
something
and
if
you
don't
personally
agree
with
it,
that's
okay,
you're
still
going
to
do
it.
This
way,
because
we
are
gonna,
go
focus
on
things
that
matter
and
I
in
particular
did
not
even
have
a
vote.
A
Novelty,
engineers,
love
new
stuff
and
sometimes
because
the
old
stuff
hasn't
satisfactorily
solved.
Our
problems,
I
totally
get
that
but
many
times
because
it's
new
we
like
to
try
new
things.
We
like
to
learn
new
things,
new
technology
new
stacks
whatever-
and
this
is
great-
but
if
we
were
scientists,
I,
so
I
love.
A
This
facial
expression
I
had
to
do
like
a
screen
capture
from
a
moving
video,
so
this
is
the
one
that
I
happen
to
get
I
think
this
is
the
way
Neil
would
look
if
he
saw
how
many
variables
we
change
when
we're
testing
something
new
like
if
you've
ever
gone
through
the
scientific
method,
you
generally
try
to
isolate
one
independent
variable.
We,
especially
in
the
world
of
microservices
I,
have
a
clean
slate.
I
can
do
all
these
things
that
I've
always
wanted
to
do
so.
Let's
change
everything
at
once.
A
This
does
not
end
well
every
single
change
compounds,
on
top
of
the
others,
to
create
basically
an
environment
that
you
know
nothing
about,
and
then,
when
you
run
into
issues
you
end
up
not
knowing
where
to
start
I
mean.
Has
anyone
here
ever
been
through
the
process
of
backing
out,
3
or
10,
or
whatever
different
changes
to
try
to
figure
out
the
one
that
mattered
right
and
you
can't
get
bisect
all
of
your
technology
stack.
A
So
a
couple
examples
of
this
and
actually
sorry
what
why
we
didn't
think
this
matter
to
us
was
we
were
adopting
technology
choices
that
actually
we
believed
we
were
familiar
with
if
that
can
make
any
sense,
but
we
we
used
massively
at
circle
CIA's.
Ever
since
2011
we've
been
a
shop
and
we
we
run
terabytes
and
terabytes
many
many
different
replica
sets.
A
We
know
it
well
and
we've
been
using
it
for
a
long
time,
but
as
we
move
to
micro
services,
we
said
okay,
this
is
an
opportunity
to
go
back
to
Postgres,
which
many
of
us
prefer.
We've
had
some
interesting
experiences
with
over
the
years,
and
so
let's
take
this
opportunity
and
it's
it's
Postgres,
it's
a
relational
database.
Everybody
knows
about
relational
databases
right
well,
it
turns
out
that
we've
actually
been
using
for
a
really
long
time,
and
so
engineers
that
have
been
working
with
us
for
a
long
time
have
been
thinking
and
living
breathing.
A
A
A
relational
database
is,
it
is
a
powerful,
powerful
tool,
but
you
need
to
understand
how
to
use
it,
and
so
now
we
are
helping
engineers
understand
these
new
old
tools
that
we're
using
versus
again
working
on
building
business
logic
day
to
day
and
I
mentioned
previously
that
we
introduced
docker,
because
we
are
moving
to
a
services
based
world.
We
have
a
bunch
of
different
deployments.
A
We
want
to
make
them
consistent
and
we,
you
know
we're
deploying
a
lot
of
it
and
we
actually
truly
believe
in
docker
I
mean
I
personally
have
worked
on
sort
of
the
operations
and
system
side
of
the
house
for
a
long
time,
maybe
20
years,
and
so
I
truly
believe
that
docker
is
a
game
changer
it.
It
really
handles
a
number
of
issues
that
I
wished.
A
I
could
have
handled
for
many
years
over
the
course
of
my
career,
but
in
introducing
it
we
also
then
brought
it
into
every
single
developers,
development,
environment
and
so
I
as
a
developer.
Instead
of
spinning
up
this
single
monolith
and
my
instance
and
getting
to
work
on
writing
code
now
need
to
manage
a
docker
environment,
a
docker
cluster,
if
you
will
or
pod
on
my
laptop
and
understand
the
interactions
between
those
and
I'm,
not
personally
a
systems
engineer
and
therefore
I,
don't
understand
when
I'm
trying
to
debug
that
this
thing
has
a
network
connection.
A
That's
not
working
to
that
thing,
and
so
my
day-to-day
job
of
just
writing
code
is
now
encumbered
with
this
new
knowledge
requirement.
So
this
seems
like
the
obvious
answer
here,
but
take
small
steps.
As
I
said,
when
we
did
the
two
dotto
project,
we
had
a
couple
engineers.
They
were
working
on
things
very
quickly
and
they
were
isolated
from
the
rest
of
the
system.
A
So,
as
we
discovered
issues,
we
were
able
to
tackle
those
issues
and
figure
out
how
to
get
around
them,
but
with
that
I
would
advise
investing
in
them,
because
when
you
realize
hey,
this
is
an
issue
for
me.
It's
going
to
be
an
issue
for
other
people,
so
you
need
to
actually
stop
and
make
an
investment
in
how
you're
going
to
tool
that
for
broader
use,
because
when
we
went
from
one
small
project
to
basically
all
projects,
everyone
was
trying
to
solve
these
things
independently
and
running
into
the
exact
same
issues.
A
A
Okay,
so
last
repetition,
we've
all
heard
it
don't
repeat
yourself.
Hopefully
you
read
this
book,
maybe
in
1999,
maybe
more
recently,
but
in
1999
Andrew
hunt
and
David
Thomas
taught
us
this
expression
dry
and
in
the
same
year,
Martin
Fowler
wrote
his
refactoring
book,
which
was
basically
how
to
break
down
your
code,
find
patterns,
code
smells
and
retool
them
appropriately,
and
if
you
look
at
the
list
of
patterns
that
he
identifies,
our
code
smells
as
he
refers
them
in
that
book.
A
The
very
first
one
is
duplication,
and
so
this
is
something
we've
been
talking
about
for
a
really
long
time.
Our
ID
'is,
if
you
happen
to
use
one,
have
built-in
functions
to
refactor
our
code
to
extract
out
common
code
to
simplify
it
down,
take
out
the
one
difference
and
make
that
into
a
single
point,
and
so
we
do
this
very,
very
well
in
the
small
right.
A
We
understand
how
to
take
our
code
and
eliminate
all
the
duplication
and
all
the
overhead
of
that
duplication
and
we've
learned
where
those
things
are
going
to
fail
us
so
I'm
sort
of
wish
that
there
would
be
a
2017
reissue
edition
of
one
of
these
books
that
doesn't
require
much
difference.
Just
has
a
little
asterisk.
Maybe
that
says
also
applies
to
micro-services,
because
we
haven't
figured
out
yet
how
to
apply
these
patterns
in
a
distributed
system.
A
So
we
didn't
think
this
was
going
to
be
a
problem
going
in
I
like
everything
else,
because
we
thought
people
would
talk
to
each
other
and
we
thought
okay.
If
someone
is
working
on
a
problem
that
they
identify
and
someone
else
is
working
on
that
a
similar
problem,
they
will
somehow
find
each
other
and
express
that
way.
A
It
does
make
it
sound
like
an
insane
idea,
but
they
will
find
each
other
and
they
will
talk
about
that
problem
and
they
will
come
up
with
a
common
solution,
and
this
makes
a
lot
of
sense
that
the
telephone
is
important
here,
because
we
actually
do
have
a
fairly
distributed
team.
We
have
a
lot
of
remote
folks
which,
which
does
create
some
additional
reduced
likelihood
that
I'm
just
going
to
bump
into
you
and
say:
hey
I'm
working
this
thing,
oh
I'm,
working
on
that
same
thing.
Maybe
we
should
get
together
and
find
a
common
solution.
A
A
Then
we
went
and
realize
that
and
took
it
back
into
those
other
two
places
and
consolidated
on
one
which
is
another
layer
of
work.
That's
not
the
work
that
you're
trying
to
do
to
get
business
functionality
in
front
of
your
customers
also
I
mentioned
moving
to
Postgres,
so
this
is
kind
of
fun.
I,
unfortunately,
don't
get
to
write
a
lot
of
code
anymore
or
fortunately,
but
it's
fortunate
for
everyone
else.
A
It's
not
as
fortunate
for
me
and
but
every
once
in
a
while,
I
like
to
skimpy
ours
and
just
kinda,
see
what's
going
on,
and
especially
as
we're
making
this
transition.
I
mean
I,
do
care
a
lot
about
how
we
are
building
things
and
whether
or
not
we're
being
effective
in
our
building,
and
so
it's
late
at
night
and
I'm
skimming
through
some
PRS
and
I
find
an
implementation
in
one
of
our
new
services
of
basically
talking
to
Postgres.
A
You
know
setting
up
the
connections,
configuring,
basically
where,
where
the
Postgres
instance
is
and
handling
retries
and
stuff
like
that
and
most
of
it
is
deferred
to
an
external
library,
and
so
it's
we'll
call
it
ten
lines
of
code.
But
it's
ten
lines
of
code.
That's
in
this
one
service
and
I
think
to
myself.
This
is
going
to
end
badly,
but
if
you're
in
engineering
management,
if
you're
a
CTO,
VP
and
whatever,
wherever
you
might
be
in
the
management
structure,
even
probably
a
you
know,
just
a
senior
engineering
with
some
experience.
A
This
is
a
really
hard
part
of
your
job,
which
is
having
the
conversation.
That
goes
something
like
you're
doing.
This
thing
and
it's
going
to
end
badly
and
that
obvious
question:
well,
how
is
it
going
to
end
badly
and
the
only
answer
you
have
is
I,
don't
know
yet
I'll
know
it
when
I
see
it
and
it
is
going
to
end
badly
well.
Why
are
you
micromanage
me?
Why
are
you
micromanaging
me
right
now,
I'm
not
I'm,
just
trying
to
help
future.
You
have
a
better
time.
You're
helping
present
me
have
a
really
bad
time.
A
A
That
is
hand
coded
it's
Postgres
connection
and
it's
all
somewhat
centrally
managed,
but
they
all
handle
the
central
management
differently.
So
now,
four
or
five
I,
don't
remember
the
exact
number
teams
not
only
need
to
go,
find
these
lines
of
code
and
find
a
way
to
solve
it.
They
each
need
to
solve
it
differently
because
their
implementation
is
different
and
they
each
need
to
stop
what
they're
doing,
because
this
was
written
months
ago.
A
They
need
to
stop
what
they're
doing
on
whatever
they're
trying
to
deliver
right
now
and
go
back
and
re
implement
their
handling
of
a
Postgres
connection.
A
Postgres
connection
is
not
very
exciting.
It's
not
moving
the
needle
for
your
customers,
I
mean
if
you
didn't
have
a
connection.
That
would
be
a
problem
for
your
customers,
but
it's
not
the
thing
that
they
are
excited
about
in
your
product
and
so
implementing
it
five
different
ways.
So
you
have
to
go
back
and
fix
it.
Five
different
ways
is
pure
overhead.
A
It
has
nothing
to
do
with
the
value
that
you're
trying
to
deliver
to
your
customers.
So
the
key
learning
that
we
took
out
of
this
particular
situation
and
others
like
it-
is
that
you
really
do
need
to
have
shared
components,
but
they
need
two
things
with
them.
One
is
you
actually
need
to
recognize
the
investment
and
create
ownership?
A
A
That's
going
to
help
our
teams
be
successful
at
building
their
micro
services
at
focusing
on
the
things
that
they
really
know
how
to
do
and
taking
away
the
things
that
are
either
going
to
be
a
big
learning
for
them
or
a
big
repetition
of
what
everyone
else
is
doing.
So
we
effectively
created
a
team,
it's
called
the
backplane
team
and
I'll.
Let
you
go
look
up.
I,
don't
know
how
blade
servers
are
architected,
which
apparently
no
one
knows
about
anymore,
but
take
a
look
at
them.
You'll
understand
what
a
backplane
is.
A
It's
all
the
shared
components,
all
the
pieces
that
basically
make
your
piece
really
really
simple
and
that's
all
they
work
on.
So
it's
not
like
someone
in
one
of
these
services
teams
says:
oh
I
need
this
thing
and
everyone
else
must
need
it.
So
I'm
gonna
build
it
and
support
it
because
now
I'm
not
working
on
my
service.
A
A
So
we
talked
about
creating
alignment
by
putting
people
in
teams,
because
communication
within
teams
is
generally
more
successful,
and
this
is
great.
But
at
some
point
you
need
communication
across
teams.
So
we
took
the
time
to
create
what
we
call
guilds,
which
is
basically
out
of
the
the
Spotify
model.
A
A
Alright,
this
is
a
look
in
the
rear
view,
mirror
except
it's
2017,
so
I
guess
we
all
have
you
know
backup
cameras.
Now
the
big
question
in
all
of
this
is
is:
was
it
worth
it
was
this
the
right
thing?
Did
we
make
the
right
decision
about
nine
months
ago
when
we
headed
down
this,
and
the
answer
for
me
is
a
resounding
yes,
absolutely
and
drawn
to
scale.
I
used
a
ruler.
A
Actually,
if
you
believe
it
to
draw
this
diagram,
but
you
know,
I
talked
previously
about
how
we
created
additional
capacity
and
we
couldn't
utilize
it.
We
actually
got
lower
and
lower
consumption
of
that
capacity,
and
so
that
system
that
that
symptom
has
gone
away.
Not
only
that
the
the
blue
line
there,
that's
where
it
tipped
over
before,
and
so
we're
running
these
days
at
about
five
times
that
utilization
of
the
platform
and
growing
solidly
and
all
of
those
operational
concerns
have
disappeared
and
we're
seeing
the
same
effects
in
terms
of
our
actual
delivery.
A
But
it
you
should
all
think
about
this.
For
yourselves,
I
don't
know
if
anyone
is
a
Cat
Stevens
fan-
or
maybe
you
just
saw
guardians
of
the
galaxy.
It's
on
the
soundtrack
anyway,
father
and
son.
This
song
is
a
great
example
to
me
of
someone
who's
been
through
something
trying
to
explain
it
to
someone
who
hasn't
liked.
Yes,
it's
easy
for
me
to
say:
oh,
these
are
all
the
things
that
we
learned,
and
these
are
the
experiences
that
we
had.
A
So
if
you're,
not
in
an
organization
that
can
have
a
healthy
conversation
about
that
investment,
then
this
map
might
might
not
be
the
right
step
to
take
minimize
communication
dependencies
again.
You
need
to
make
a
decision
in
your
organization
about
where
you
want
the
fast
paths
of
communication
to
be
Conway.
We
talk
about
all
the
time.
It
really
is
going
to
have
an
impact
on
how
you
build
and
where
your
successes
are
going
to
be
so
think
about
that
a
lot,
but
then
make
sure
you
do
your
best
to
get
that
cross
team
communication.
A
We
love
the
notion
of
micro
services
being
an
opportunity
for
complete
autonomy.
You
go
build
your
service
in
your
little
locked
room.
I'll
go
build
my
service
in
my
little
locked
room
and
maybe
we'll
share
an
API
spec
and
nothing
else,
and
this
might
be
right
for
you,
depending
on
the
scale
of
your
organization,
but
at
the
scale
of
our
organization,
we're
about
30
software
developers.
This
does
not
make
sense.
A
We
need
to
have
clear
communication
paths
so
that
we
can
get
those
points
of
leverage
in
sharing
and
then
reduce
decisions
to
exceptions
get
together
early
decide
how
you're
going
to
do
things,
obviously
I'm
pushing
for
as
much
commonality
and
sharing
as
possible,
but
make
that
decision
and
only
choose
to
do
other
things.
When
that
decision
completely
fails,
you
don't
choose
my
sequel
because
I
like
it
better
choose
my
sequel,
actually,
don't
choose
my
sequel
for
using
Postgres,
there's,
no
good
reason,
but
maybe
you
need
a
graph
database
to
solve
a
very
specific
problem.
A
Also
think
you
probably
don't
talk
to
me
after
you
wanna
hear
about
graph
databases,
but
only
choose
to
even
think
about
alternatives.
If
you
truly
can't
make
the
path
that
you've
set
work,
because
every
new
thing
you
introduce
is
gonna
be
overhead
and
every
time
you
stop
to
evaluate
that's
work
that
you
shouldn't
be
doing,
because
you
could
be
delivering
functionality
of
your
customers
and
then
finally
iterate
so
nine
months
ago
we
sort
of
embarked
on
this
journey
and-
and
we
thought
we
were
right-
we
thought
we
had
the
right
organization
in
place.
A
We
thought
we
knew
what
we
were
doing.
We
we
looked
at
these
particular
problems
that
we
thought
we
might
run
into
and
and
we
thought
we're
gonna
handle
them.
So
we've
done
a
lot
to
change.
Over
that
time.
We've
created
a
new
organizational
structure.
We've
thought
about
how
we
take
on
problems
differently,
but
the
most
important
thing
that
I
think
we've
taken
out
of
that
is
that
our
architecture
is
constantly
changing.
We
have
become
very
good
as
a
community
at
iterating
on
the
software
that
we
build
right.
We
try
to
get
very
early
feedback.
A
We
use
circle
CI
for
our
CI
NCD.
We
get
stuff
in
front
of
customers
very
quickly.
We
learn
and
we
iterate,
but
if
we
don't
do
that
with
our
organization,
then
our
software
will
continue
to
move
in
new
directions
and
we
won't
be
aligned
with
it
anymore.
So
it's
not
about
setting
up
your
organization
in
a
way
that
matches
what
you
do
today.
It's
about
setting
up
your
organization
in
a
way
that
can
respond
to
the
ever
changing
state
of
your
software
and
architecture.