►
From YouTube: The Spotify Story: Moving from NIH to an Active CNCF Community Member - Jim Haughwout, Spotify
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
The Spotify Story: Moving from NIH to an Active CNCF Community Member - Jim Haughwout, Spotify
Four years ago, at Spotify, we built and managed the majority of our infrastructure ourselves. We managed our own data centres. We built our own time series database. We built our own orchestration technology (ironically releasing it the same week as Google did Kubernetes). Today our world is very different. We serve over 200 million monthly users from a cloud provider. We are a CNCF member. We have more than doubled our open source initiatives in the last two years. In this talk, Jim Haughwout--Head of Infrastructure & Operations at Spotify--will talk about Spotify’s path on this journey, including how it’s “Build Infrastructure Together” strategy is helping build infrastructure faster and lower adoption costs. He will also share the challenges has Spotify faced on this continuing journey, and some early learnings that may help other CNCF member companies.
https://sched.co/MPZY
A
Hello,
everyone
welcome
to
the
this
session
of
the
community
track.
Next,
speaker
is
Jim
Hobart.
He
is
the
head
of
infrastructure
and
operations
at
Spotify
and
he
will
talk
to
us
about
how
Spotify
became
an
active
community
member
of
CNCs
and
at
the
end
of
the
talk,
please
don't
forget
to
go
on
scared,
calm
and
rate.
The
talk
thank.
B
B
B
B
B
So
restart
so,
as
mentioned,
my
name
is
Jim
I
lead
infrastructure
and
operations.
We
call
that
IO
for
short
at
Spotify.
Essentially
that's
the
core
infrastructure
team
for
part
of
our
platform
mission
at
Spotify
and
the
job
of
platform.
Is
we
get
up
every
day
and
we
can
ask
about?
How
can
we
make
our
engineers
basically
build
code
faster
scale,
faster
scale
safely
and
basically
be
an
amplifier
for
productivity
at
Spotify?
B
So
today,
I'm
going
to
talk
about
our
move
from
basically
and
not
invented
here:
do
it
ourselves
infrastructure
provider
to
act,
an
active
community,
member
and
I,
put
her
in
more
active
community
member,
because
we're
still
growing
and
that
were
been
inspired
by
lots
of
you.
This
is
a
talk
about
ways
to
move
lots
of
people
who
are
really
busy
who
are
using
thousands
of
existing
services
with
lots
of
existing
infrastructure,
libraries
and
dependencies
lots
of
traffic
lots
of
competing
priorities.
B
And
how
do
you
go
from
in-house
technology
on
the
path
to
open,
CN,
CF
and
other
Foss
technology
and
before
I,
get
started?
We're
still
on
this
journey
by
our
account
we're
on
year,
2
of
it.
From
that
perspective,
I'll
talk
a
little
about
that.
We
haven't,
found
all
the
answers.
However,
we
we've
made
some
mistakes.
B
We've
had
some
interesting
lessons
that
we've
learned
and
this
community
has
been
so
tremendously
helpful
to
us
that
it's
my
hope
that
sharing
some
of
this
will
help
those
of
you
who
are
trying
to
go
back
and
help
your
organizations
move
to
kubernetes
and
G
RPC
and
the
tests
and
everything
else.
So
so,
let's
get
started
from
that.
So
a
little
bit
about
Spotify.
You
all
know
it's
a
streaming
music
provider,
2-sided
marketplace
for
consumers
and
artists,
but
I
wanted
to
talk
a
little
about
our
infrastructure.
B
We
love
infrastructure
at
Spotify,
it's
the
heart
of
our
technology.
It's
actually
one
of
our
oldest
technology
teams.
We've
been
an
early
adopter
of
modern
infrastructure,
like
micro
services
containers,
some
of
the
people
sitting
in
the
front
row
actually
join
Spotify
because
some
the
demos
they
saw
some
of
our
doctor
meetups
we've
extended
the
concept
of
micro-services
into
web
sites
and
data
pipelines.
We're
actually
now
looking
and
doing
that
into
micro
features
for
our
client
architecture,
but
from
the
purpose
here
it's
important
to
know
is
we
don't?
We
have
autonomous
teams
that
are
unblocked.
B
We
do
not
have
a
central
ops
group
that
controls
releases.
Every
team
manages
their
own
software
controls.
Their
releases
can
pick
up
technology
or
on
their
own.
So
this
is
a
discussion
of
how
do
you
change
hundreds
of
teams
with
thousands
of
services?
From
that
perspective,
that
point
of
view-
and
that
was
kind
of
our
challenge
a
little
bit
about
us-
we
built
a
lot
of
her
own
infrastructure.
It's
by
necessity.
We
actually
started
doing
this
six
seven
years
ago
before
there
was
CN
CF.
B
Basically
it
was
very
cool.
Yesterday
they
here
dan
call
out
helios
as
one
of
anarchist.
That's
our
container
orchestration
technology,
which
we
as
people
like
the
joke.
We
announced
one
day
before
kubernetes
was
announced.
Clearly
one
of
the
two
has
a
large
number
of
contributors
going
forward
and
that's
part
of
the
discussion
today,
but
basically
that's
how
ingrained
infrastructure
is
to
our
DNA.
These
are
some
of
the
projects
up
here.
I,
don't
have
enough
time
to
go
through
all
them.
You
can
find
them
all
in
github.
B
So
take
a
look
afterwards,
there's
also
not
on
here.
Our
most
popular
project
is
called
Luigi.
It
doesn't
have
a
CNC
F
equivalent.
It's
a
data
pipeline
Orchestrator.
It's
got
a
bit
11,000
stars
on
github,
so
check
that
out
as
well,
but
essentially
we've
been
using
these
technologies
at
scale
for
years.
Here
are
some
numbers.
There
are
all
public
numbers
that
you
may
have
seen
in
presentations.
We
put
them
in
one
place.
We
have
two
over
200
million
monthly
active
users.
We
have
over
2,500
micro
services
that
were
basically
moving
to
new
technologies.
B
At
the
perimeter
we
run
a
peak
at
between
8
&,
9
million
QPS.
So
from
that
perspective
we
capture
billions
of
events
per
hour
that
we
use
for
everything
from
machine
learning
to
paying
artists.
We
do
hundreds
of
deployments
and
commits
a
day,
and
we
have
currently
220
open
source
projects,
at
least
as
of
Monday
on
github.
You
can
take
a
look
at
from
that
perspective,
so
a
big
question
is
we're.
B
If
we're
doing
over
a
thousand
deploys
a
day,
if
we've
got
all
these
micro
services,
we've
been
on
containers
for
years,
why
would
we
want
to
change
anything?
Why
are
we
making
this
jump?
You
know
why
make
the
change
and
the
really
simple
answer?
Is
it's
the
power
of
community?
We
have,
you
know
basically,
our
own
infrastructure
teams,
but
there's
no
way
that
our
teams
alone
can
compete
with
the
world
at
large.
We
believe
by
joining
a
community,
that's
organically
growing.
We
can
benefit
from
their
pace
and
volume.
B
We
can
give
back
based
on
our
expertise
and
I,
had
some
interesting
slides
on
this,
but
yesterday
Jenna,
basically
Cheryl
and
Janet
really
did
this
justice,
so
fifty
six
thousand
contributors
I
think
that's
bigger
than
any
infrastructure
team
and
any
company
around
the
world
and
that's
in
five
years
and
that
kubernetes
saw
4x
increase
after
they
want
to
see
ncf.
That
really
speaks
to
the
power
of
community
that
we're
all
in
this
together
we're
solving
the
same
infrastructure
problems.
We
have
different
edge
cases,
we
have
different
scales
for
us.
B
It
would
be
really
easy
to
still
stay
and
Hermes
and
Helios
and
keep
adding
things
incremental,
less
change
and
coordination
across
our
company,
but
we
know
that
that's
not
the
right
way
to
go.
The
right
way
to
go
is
to
tap
those
56,000
people
and
contribute
to
that
as
well,
and
the
trick
is
how
do
we
do
that
at
scale
at
the
paces
that
we
showed,
if
out,
adding
too
much
risk
or
delay
so
a
little
about
what
we
did
first,
so
from
the
purpose
of
time?
This
is
about
the
last
two
years.
B
So
the
very
first
thing
is
we.
We
joined
lots
of
communities
from
that
case.
Some
of
these
communities
were
members
in
and
we
leveled
up
or
involvement.
Others
we
joined
for
the
first
time,
so
we
joined
CN
CF
about
15
months
ago.
From
that
perspective
and
realize
joining
and
talking
to
others
was
the
key
step.
The
next
thing
is,
we
started
actively
working
with
CN,
CF
and
other
faust
technologies
and
point
out.
This
is
not
just
hack
days.
B
We
had
whole
teams
of
engineers
who
were
working
on
this
full
access
to
our
cloud
and
at
the
time,
on-prem
resources
now
we're
100%
in
the
cloud
from
that
perspective,
but
it
was
a
full
teams
of
product
managers
of
engineers,
heavy
investments,
working
on
this
finding
the
edge
cases.
What
worked?
What
didn't
trying
things
out?
We
even
started
building
some
things
completely
in
the
open.
B
We
have
several
people
that
are
big
Foss
advocates
in
a
variety
of
communities,
both
client
back
in
multiple
languages,
and
we
actually
started
build
projects
completely
in
the
open
from
day
one.
So
every
commit
was
visible
to
all
of
you.
We
had
already
had
many
projects
that
were
open
before,
but
this
was
taking
a
deep
dive
into
that.
So
things
first,
things
started
off
really
well,
one
is
people
were
really
excited
about
the
change.
There
were
some
concerns.
We
had
spent
years
building
all
of
our
own
infrastructure.
B
Would
we,
you
know
basically
disincentivize
people
to
stay?
Would
it
be
like
I'm,
really
passionate
about
Helios
and
I?
Don't
want
to
move
to
kubernetes
or
I.
Don't
want
to
move
the
G
RPC
when
people
feel
like
we
weren't
special
anymore
and
that's
a
question.
I
talked
to
so
many
companies
that
people
are
concerned
about
and
we
found
the
exact
opposite
engineers,
love,
learning
and
trying
new
things.
B
So
people
were
excited
about
trying
great
technology
finding
or
it
was
broken,
making
it
better
learning
things
that
were
new,
taking
our
lessons
and
helping
out
others
zero
employees
left
because
we
made
these
changes.
In
fact,
a
lot
of
other
people
join
Spotify
and
a
lot
of
people
from
non
infrastructure
teams
heard
about
what
we're
doing
and
came
over
and
joined
the
teams,
for
that,
so
don't
be
afraid
if
you're
thinking
about
it.
B
B
We
found
our
culture
starting
the
change
in
subtle
ways
and
just
by
think
looking
at
how
other
people
were
doing
things
and
some
of
the
challenges
and
some
of
the
design
approaches,
push
versus
pull
and
the
like
got
us
to
rethink
some
of
our
engineering
and,
as
basically
like
one
tangible
example
is
ephemerality,
is
something
we
had
talked
about,
but
never
heavily
are
protected.
We're
now
architecting
for
ephemerality,
everywhere,
from
reliability,
to
tracing,
to
observability,
to
storage
and
actually
taking
a
concept
like
that.
B
So
we
can
harness
kubernetes
and
make
it
available
and
working
in
every
part
of
our
architecture
and
that's
something
it
was
a
deeply
enriching
because
it
just
brings
so
many
new
ideas,
and
so
many
new
engineers
in
and
we
continuously
get
better
and
the
next
one.
Basically,
we
talked
about
this
last
year
at
one
of
our
keynotes
is
we've
been
amazingly
impressed
about
the
generosity
of
this
community,
so
many
people
reached
out
and
shared
ideas
with
us.
We've
tried
to
share
lessons.
Learning
back
as
well.
B
B
And
it's
a
at
last
minute
they
often
came
and
helped
out,
and
that
is
something
we've
seen
across
CN,
CF,
I
wouldn't
say:
CN
CF
is
an
awesome
community
I'd
say
it's
a
community
of
awesome
people
that
work
on
that.
So
that's
just
one
of
the
lessons
and
if
you
haven't
paid
your
end
user
dues,
yet
they're
worth
doing,
you'll
learn
a
lot.
So
it
sounds
like
everything
is
amazing.
B
Was
it
not
exactly
once
we
got
deeper
into
things?
We
hit
a
few
brick
walls
and
in
fact
we
hit
several
brick
walls,
so
I
want
to
share
some
of
the
painful
things
that
we
learn
from
this.
So
the
first
one
is
plugging
things
into
an
ecosystem
is
really
hard.
I
could
say
this
is
true
for
really
big
companies
with
lots
of
micro
services,
but
I've
talked
to
people
who
are
you
know.
B
I've
spent
ten
years
in
startups
and
three
engineers
and
plugging
into
things
and
doing
upgrades
is
really
hard
there
as
well,
especially
when
you're
doing
on
call
during
at
night
and
development.
During
the
day.
It's
it's,
the
bigger
you
are
the
more
moving
parts
there
are.
So
you
know,
basically,
we
have
lots
of
edge
cases.
B
It
was
part
of
our
move
to
cloud
we
talked
about
that
last
year.
In
other
conferences,
we
have
providers
that
help
us
manage
some
of
our
services
for
us,
so
we
can
free
up
engineers
the
tackle
unique
challenges
to
us.
As
we've
worked
with
those
providers.
Given
our
scale
on
complexity,
we
had
certain
needs.
So
how
did
we
get
them
to
build
the
features
and
capabilities
that
match
our
scale
of
complexity
and
does
that
get
rolled
back
into
the
CN
CF
project?
Or
is
that
in
the
managed
service
project?
B
So
that's
the
next
and
then
the
last
is
adding
all
of
our
own
integrations.
Everyone
in
this
room
has
services
that
are
integrated
with
everything,
from
logging
to
data
pipelines,
to
reporting
to
everything
the
other
CFO
needs
to
ensure
our
customers
internally
had
everything
that
they
had
before.
Moving
the
technologies
like
Nettie's
or
g
or
pc,
we
had
to
do
a
lot
of
integration
into
our
data
pipelines,
things
that
give
us
competitive
advantages
for
music
recommendation
we
had
to
plug
into,
and
the
like
and
just
controls
to
say
compliant
as
a
business.
B
So
this
only
at
that
point
is
something
ready
for
adoption,
so
that's
kind
of
the
magic
sweet
spot
when
all
three
of
those
coming
together.
One
thing
is
this
takes
a
lot
of
work
and
it
takes
a
lot
of
time
and
some
technologies.
This
took
18
months
for
us
to
do
to
work
and
roll
out
I'll.
Give
you
a
really
simple
clear
world
example
for
kubernetes.
First,
we
had
to
basically
have
a
shared
virtual
private
cloud
so
that
all
of
our
individual
projects
that
work
together-
we
have
many
many
namespaces.
B
B
Then
we
had
to
connect
everything
into
our
ecosystem,
so
we
could
do
everything
from
paying
artists
to
do
music
recommendations
and
the
people
can
move
from
Helius
to
a
brunet
ease
without
any
loss
of
features,
and
we've
now
finished
that
we're
seeing
migration,
it
would
have
been
easier
to
just
keep
it
incrementing
features
that
would
have
been
a
lot
less
costly.
A
lot
simpler,
but
we
knew
the
right
thing
to
do
is
to
get
on
the
kubernetes
train,
because
the
echo
system
we
plug
into
this
took
time,
but
our
customers
are
very
happy.
B
That's
we've
learned
migration
as
a
four-letter
word,
and
here's
a
few
reasons
why.
First,
everyone's
busy
infrastructure
teams
are
busy
and
infrastructure
feature
teams
are
busy
on
features.
Data
teams
are
future
and
data
data.
Everyone
has
lots
of
priorities
asking
someone
to
migrate,
just
because
it's
better
yeah,
it's
not
good
enough.
Basically,
that
leads
us
into.
We
have
to
explain
why.
B
Basically,
you
know,
why
should
I
pick
this
technology
first
another?
What
do
I
save
by
doing
this?
Getting
exploiting
the?
Why
gets
feedback
gets
people
excited,
for
example,
in
kubernetes
we
had
to
explain
why
kubernetes
first
and
not
serverless,
later
first,
and
instead,
why?
For
us
service
list
server
list
makes
sense
to
come
later.
The
others
be.
You
know,
cloud
providers
have
application
engines
as
a
service.
B
Why
don't
I
just
use
that,
like
other
companies
versus
kubernetes,
so
we
had
to
kind
of
tell
that
story
to
get
people
on
board
before
they
could
prioritize
the
work
to
do
adoption
and
then
the
third
lesson
gets
into
there's
many
more
degrees
of
freedom
than
you
imagine.
In
our
field
we
like
to
use
the
analogy
of
I'm
changing
the
engine
in
flight
and
the
plane
didn't
crash,
I'm,
an
aerospace
engineer,
so
I've
always
loved
this
analogy,
but
we
found
especially
in
a
world
of
unblocked,
asynchronous
teams.
This
is
way
too
simple.
B
B
That's
we
found
that
something
that's
been
able
to
tell
the
story
and
help
line
up
people,
so
it's
much
easier
for
them
to
migrate
and
adopt
technology.
The
last
building
in
the
open
is
is
hard.
We
actually
did
this
a
little
earlier
than
we
were
ready,
so
I
kind
of
like
you
would
what
we
wouldn't
go
out
and
run
one
day
and
then
run
a
marathon.
You
train
for
a
while,
when
you're
building
the
open,
every
pull
requests.
B
You
start
to
a
question:
they
did
enough
test
coverage,
it's
my
documentation,
clear
as
they
pull
requests
even
written
nicely
from
that
perspective,
so
that
observer
effect
can
slow
you
down.
So
we
had
to
learn.
You
had
to
build
those
muscles
to
be
able
to
build
openly
and
we
learn
basically
starting
slower
and
then
going
open
source
later.
So
we
hit
all
these
for
about
nine
months.
We
found
this
kind
of
confounding
we're
like.
Should
we
actually?
Is
this
just
too
hard?
Should
we
just
go
back
and
increment
things?
We
had
a
ton
of
priorities.
B
B
We
were
hitting
some
walls
and
he
said
what
should
we
do
and
lesson
came
basically
from
our
teams
and
from
our
community,
and
it
was
a
really
the
groups
that
were
building
the
most
community
inside
and
outside
of
Spotify
found
the
answer
and
what
it
came
down
to
was
here's
a
little
bit.
I
can
tell
a
little
bit
of
the
story
on
this
is
four
years.
We
do
something
called
embedding
so
people
come
and
they
join
a
team
and
they
work
with
that
team
for
a
while
and
they
try
it
out
and
they
like
it.
B
B
And
basically
this
solved
several
problems.
We
actually
knew
we
were
solving
our
customers
problems.
We
also
got
some
extra
resources
to
solve
things,
and
then
we
built
ambassadors
who
could
go
back
into
their
teams
and
educate
others
how
to
use
the
technology.
So
basically,
this
was-
and
this
also
allowed
us
to
accelerate
focusing
on
things
that
had
the
most
important
need.
So
we
could
solve
the
biggest
problems
from
that
perspective,
and
essentially
you
know
this
led
to
a
really
important
learning.
Is
that
the
path
the
community
is
not
the
jump
outside?
B
First,
it's
to
being
more
open
internally.
First,
it
gives
you
the
safe
space
to
build
those
muscles
to
work
with
others
to
actually
have
more
people
than
your
team
on
a
project.
From
that
perspective,
to
think
about
your
documentation
to
think
about
the
level
of
test
coverage,
it's
that
safe
environment
and
you
get
benefits
in
less
time
and
implying
this.
The
results
are
pretty
amazing,
so
these
curves
are
anonymized
curves
of
various
infrastructure
components
that
have
had
adoption
over
the
last
15
months.
B
Some
of
them
are
up
at
80%
adoption,
some
like
you've,
seen
in
some
the
presentation
say
around
10
or
11%,
but
those
are
0
to
10%
and
anywhere
from
60
to
90
days.
The
biggest
driver
of
this
is
when
we
worked
with
others
and
they
adopt
the
technologies.
They
would
go
back
and
brag
about
this
really
cool
thing.
They
did
and
it's
not
you
saying
the
infrastructure.
It's
awesome.
It's
your
customer
saying
this
solved
the
huge
problem.
B
We
can
go
five
times
faster
now
and
builds
and
then
once
other
customers
would
see
that,
like
me
and
then
they
joined
and
we
built
together
and
that
valid
and
that
basically
validate
our
beliefs
and
two
things
in
community
and
unblock
teams
from
that
case,
letting
teams
kind
of
pick
that
direction.
So
this
led
us
to
adopt
a
very
purposeful
strategy
on
this
that
we've
invested
very
heavily
and
our
infrastructure
teams
and
our
platform
at
large.
B
We
call
this
building
infrastructure
together
and
basically,
when
flushing
out
the
strategy,
it
has
three
particular
ways
of
how
we
build
things.
The
first
is,
rather
than
inventing
things
that
exist,
we
look
first
in
the
community
and
we
say:
is
there
a
cloud
natives?
Is
there
a
CNCs
solution?
Is
there
another
fall
solution?
Is
there
a
cloud
native
partner
has
a
solution,
and
we
say:
let's
look
at
that.
First,
it's
not
ready.
Let's
contribute
to
it.
Let's
see
it
could
be
pull
requests,
it
could
be
issues
it
could
be
designed
discussions
by
making
it
better.
B
It's
better
for
us,
it's
better
for
others,
and
by
the
way
we
can
pick
up
all
the
lessons
and
edge
cases,
they've
already
learned,
which,
as
an
engineer,
you
never
think
about
the
very
beginning
when
you're
building
something
the
second
one
is
building
with
others.
So,
rather
than
our
team
going
off
and
building
things
in
isolation,
we
basically
are
building
as
much
as
possible
with
others,
and
that
could
be
customers
that
could
be
partners
that
could
be
people
in
this
room.
B
We've
had
some
discussions
about
that
and
that
allows
us
to
take
our
expertise
and
domain
knowledge
and
other
people's
dirty
sand
domain
knowledge,
and
in
some
cases
we
may
not
have
enough
resource
and
someone
else
made
out
of
resources.
But
together
we
can
tackle
something
and
it
creates
natural
ambassadors
between
teams
for
sharing
where
lots
of
innovation
happens,
and
then
the
third
one
is.
We
encourage
others
to
build
on
us.
So
we've
shifted
much
more
purposely
to
platform
izing
our
technology,
and
this
is
not
just
api's
and
SDKs,
but
it's
building
platform.
B
So
people
can
add
plugins
it's
documentation,
so
people
can
understand
technology
without
calling
us.
But
it's
also
setting
up
the
false
tile
things
like
slack
channels
and
Stack
Overflow
pages
and
the
like,
so
people
can
do
things
themselves
and
you
know
essentially
staffing
or
what
we
call
goalies
and
our
slack
channels
to
answer
questions
from
that
perspective.
But
the
idea
to
make
our
technology
easy
enough.
The
people
can
pick
it
up.
B
Just
like
they'd
pick
up
an
open
source
project
and
with
that
basically
we
we
see
fulfillment
of
use
cases
we
don't
think
of,
but
someone
else
needs
to
and
it
enriches
our
platform,
and
this
is
not
just
a
collaborative
development
effort.
One
of
the
lessons
is
a
collaborative
go
to
market
planning.
So,
as
I
mentioned
before,
that
was
a
big
learning.
B
We
were
just
saying:
hey
here's
something
new
to
pick
up
for
infrastructure,
and
then
our
data
teams,
like
here's
something
new
for
data
and
our
client
teams
like
here's,
something
new
for
client
and
one
of
our
customers
said:
here's
the
900
emails
that
I
got
in
the
last
year.
Asking
here
are
new
things:
instead
we're
starting
to
work
together
and
have
a
common
roadmap
and
a
story
that
tell
our
customers
and
external
stakeholders
here's
what
we're
producing
and
we
work
with
our
customers
to
say
what's
the
best
time
to
adopt
things.
B
So
we
actually
now
pull
things
together.
We
may
delay
items
so
that
it
makes
it
easier
for
people
to
pick
up
things
to
work
with
us
to
pick
them
up
to
give
us
feedback
from
that
perspective,
and
that
is
a
big
shift
that
we've
had,
and
here
this
this
graph
is
about
a
week
old.
It's
mentioned
this
strategy
that
we
formally
adopted
is
100
189
and
do
the
things
we
like
to
look
at
you're,
looking
good
buzz.
What
are
pull
requests,
so
this
is
one
of
our
more
popular
services.
B
The
area
in
the
box
is
what
happened
after
introducing
the
strategy,
and
you
can
see
that
the
massive
increase
in
pull
requests
now
3/4
of
the
pull
requests
come
from
outside
of
our
internal
community
from
that,
so
we're
reaping
the
benefit
from
that
perspective.
So
this
we're
feeling
really
good.
We
still
had
one
nagging
doubt
and
it's
one
of
the
things
that
kind
of
you
know
keeps
our
infrastructure
leaders
up
at
night.
We're
doing
all
this,
but
are
we
actually
investing
in
the
right
technologies?
Is
it
kubernetes
versus
serverless?
B
You
know
which
proxy
are
we
going
to
use?
What
are
we
doing
for
service
mesh?
There's
been
enough
conversations
about
that
in
the
last
few
days
and
trying
to
guess
where
to
invest
can
be
really
dangerous.
You
make
the
wrong
investment.
You've
got
to
retool.
How
do
we
select
where
to
put
things?
How
do
we
avoid
missing
out
something
brand-new?
And
you
know
looking
last
year,
some
wordly
had
a
great
kamar
presentation
and
keynote
about
winning
the
battle
and
losing
the
war
he
weren't
focusing
on
the
right
thing.
B
So
we're
like
how
do
we
have
a
strategy
that
tackles
this
and
after
wrestling
with
this
for
about
4
months,
we
came
to
the
realization,
as
we
already
actually
had
the
strategy
in
place.
It
was
looking
at
the
building
infrastructure
together
strategy
and
a
very
different
point
of
view,
and
it's
one
about
community.
B
So,
basically,
when
we
kick
off
our
efforts
about
18
months
ago
and
we
started
joining
communities,
we
view
this
as
a
point-to-point
journey,
we're
going
from
do
it
ourselves
not
invented
here,
to
embracing
open
technologies
and
along
the
way
we
hit
some
turbulence.
We
basically
needed
to
stop
and
refuel.
We
got
stuck
and
we
viewed
this
building
infrastructure
together
as
a
waypoint.
We're
gonna
stop
here.
We're
gonna,
refocus
rest
build
some
muscles,
so
we
can
build
in
the
open
more
easily
and
then,
after
a
while,
we
realized
that
building
infrastructure
together.
B
Was
that
estimation,
because,
if
you're
always
working
if
the
community,
if
you're
always
working,
if
your
customers,
if
they're
always
working
with
you,
if
there's
some
new
great
idea,
you're
gonna
find
out
really
soon
you're
gonna
work
with
others
and
adapt
that
really
soon
you're
gonna.
Try
it
out
and
find
the
edge
cases
and
say:
is
it
worth
it
or
is
it
not
worth
it
and
by
actually
always
building
with
others?
We
have
a
much
lower
risk
of
missing
out
in
the
next
big
thing
and
we
stay
current
basic
permanently
connected
in
community.
B
So
now,
building
with
others
is
our
destination.
It's
what
we're
doing
right
now
and
it's
where
we're
going
forward
from
this.
So
essentially,
what's
next
we're
in
year,
two
of
this
journey
and
you're
one
still
of
building
in
fertile.
What
we'd
like
to
do
is
just
basically
do
more
and
do
better
what
we're
doing
so,
increasing
our
involvement
with
open
communities,
more
pull,
requests,
more
design
discussions
building
more
in
the
open.
We
have
a
second
team.
Now,
that's
in
infrastructure
and
operations,
it's
building
fully
in
the
open.
Now
sharing
lessons
from
the
first.
B
We
want
to
formalize
our
work
with
others.
We've
had
some
preliminary
discussions,
but
we
would
love
for
someone's
doing
interesting
things
in
service
measure
someone's
doing
in
finishing
things
and
observability
and
they're
wrestling
us
something.
Maybe
we
work
together
and
we
solve
that
together
and
instead
of
you
know,
we
both
win
and
we
both
come
out
with
that
solution
and
maybe
co-present
in
November
from
that
perspective,
at
CNCs,
North
America
and
our
long-term
aspiration
is
to
have
our
own
top
level
projects.
B
We're
really
inspired
by
the
work
that
many
of
you
are
doing,
people
that
are
smaller
than
us
that
have
open
top-level
projects.
I
look
at
lift
has
had
a
busy
business
like
us,
and
the
Envoy
has
been
something
we
use
and
many
others
and
has
been
a
big
inspiration.
Shopify.
Oh,
we
exchanged
a
lot
of.
They
have
twice
as
many
open-source
projects
at
us
as
us,
so
that
Intuit
yesterday
got
the
award
and
I
watched
that
they
got
the
2500
micro
services
on
kubernetes
in
a
year.
So
that's
a
it's
a
challenge
accepted
by
us.
B
If
you
want
to
collaborate,
reach
out
to
me,
reach
out
to
any
spot
of
fire,
we're
looking
to
people
to
build
things
together
with
we're
also
launching
a
new
speaker
series
that
will
be
online
internal
and
external
speakers.
People
who
have
great
stories
to
come
share
what
they're
doing
if
our
engineers-
and
perhaps
we
can
use
our
channels
to
augment
your
stories
as
well
so
come
take
a
look
at
that
and
if
you
would
like
us
to
come
visit
you
and
share
it,
we're
doing
we're
happy
from
that
as
well.
C
C
Hello
I
would
like
to
know
so
you,
if
you
do
develop
in
the
open.
So
as
a
company,
you
have
some
vision
for
the
software
component
right
and
how
do
you
deal
with
if
there
is
a
contradictory
development
from
what
pull
requests?
Basically,
yeah
in
your
project,
I
mean
there's
the
end
of
question
yeah,
so.
B
It's
the
question
is
like:
how
do
we
deal
if
we're
doing
a
pull
request
and
there's
contradictory
paths
in
part,
because
we're
we've
had
autonomous
teams
for
years?
That's
something
that
we're
used
to
having
those
kind
of
debates.
Often
what
we'll
do
in
those
cases
is
you
know
people
will
tend
to
pilot
both
of
their
own
systems.
We
might
even
have
a
fork
or
a
branch
or
something
or
two
separate
systems.
D
B
Yeah,
so
it's
basically
mentioned
the
having
many
different
baselines
and
many
different
technologies,
and
how
do
we
do
that?
We
have
internally
something
called
like
a
golden
path,
and
essentially
these
are
the
set
technologies
that,
if
you
choose
to
use,
we
hope
and
believe
things
will
be
easier
from
that
perspective
and
we
aspire
every
day
for
that
within
each
of
our
teams
that
are
building
infrastructure.
We
have
a
basically
a
product
manager
and
we
treat
the
infrastructure
as
a
product.
They
do
things
like
customer
advisory
boards.
They
share
public
road
maps.
B
If
you
go
to
the
project
pages,
we've
now
invested
in
basically
using
having
good
documentation
and
crawl.
All
of
our
documentation
is
converted
the
static
HTML.
So
you
can
search
on
that
and
what
we
do
as
part
of
our
storytelling
is
twice
a
year.
We
go
out
and
we
say:
here's
our
road
map,
here's
what
we're
doing
this
quarter,
here's
what
we're
guessing
next
quarter
and
here's
future,
because
we
are
an
agile
team
and
it's
hard
to
predict
past
that
and
then,
as
we
do
basically
monthly
and
quarterly
calls
of
our
customer.
B
So
we
share
about
big
things
that
are
going
out
and
by
engaging
with
customers,
and
we
actually
also
have
internal
developer
advocates
and
many
of
the
organizations
that
embed
and
get
information
back.
We
try
to
use
that
communication
to
juggle
schedules
in
such
a
way
that
things
are
less
painful.
It's
not
perfect,
but
it's
one.
That's
gotten
demonstrable
better
than
we've
been
doing
this
for
about
18
months.