►
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Oops!...I Did It Again — I Reinvented gRPC - Austin Lamon & Dave Zolotusky, Spotify
Does your organization have talented engineers who know the in’s and out’s of your infrastructure and operations? Do they know just the tool that will solve all your problems and want to build it? That happens every day at Spotify and is the reason we’ve built our own RPC framework… complete with it’s own CLI, protocol, rate limiting, circuit breaking, and performance tooling... that we're actively moving away from toward gRPC. Metaphorically, it’s much like choosing to build the perfect puzzle piece to fit within existing infrastructure versus tweaking infrastructure and adopting an Open Source puzzle piece. In this session we will discuss this ‘reinvention’ phenomenon, explain the tradeoffs of building foundational technologies like gRPC in-house (we accidentally built a Kubernetes competitor many years ago too, oops!), and detail the often overlooked benefits of using Open Source.
https://sched.co/MPZk
A
Much
for
coming,
it
seems
about
half
the
people
walked
in
with
Rolly
bag,
so
we're
close
to
the
taxi
line
on
the
way
out,
so
I
promise
stay
for
the
whole
thing.
It's
worth
it,
but
thank
you
all.
So
much
for
coming
and
I
hope.
You've
had
a
fabulous
few
days
at
Q,
Khan
and
and
where
we're
hoping
to
continue.
That
trend
sharing
some
stories
from
Spotify
and,
in
particular,
a
use
case
around
your
PC.
A
A
First,
I'll
start
by
introducing
us
I'm
Austin,
and
this
is
Dave.
We
build
infrastructure
at
Spotify.
Basically,
we
build
the
boxes,
the
tens
of
thousands
of
services
go
in
and
all
the
lines
on
the
architecture
diagrams
that
connects
them
and
basically
figure
out
how
that
all
gets
to
the
back-end
data
stores
and
make
all
of
that
work.
A
C
A
Just
say
it's:
it's
not
quite
that
simple
and
things
get
a
little
bit
complicated
in
particularly
for
Spotify.
It's
it's
everywhere,
and
hopefully
all
of
you
use
it,
but
what
we
have
not
just
kind
of
the
mobile
apps
and
desktop
apps
that
you
think
about,
but
there's
a
lot
of
hardware
that
it's
out
there
running
on
a
150
plus
speakers,
Smart
TVs,
smart,
other
smart
devices
running
across
70
plus
different
countries.
A
Some
of
them
was
slower,
maybe
not
LTE
connections
and
in
figuring
out
how
we
get
all
of
this
heavy
data
to
customers
really
really
quickly.
Is
that
it's
a
big
and
exciting
challenge
for
us
and
hey:
look
buzzword,
CN,
CF
technologies.
We
use
a
lot
of
them
and
we're
so
excited
to
be
big
contributors
to
them
and
an
evangelist
for
the.
But
these
technologies
are,
at
the
end
of
the
day,
a
means
to
an
end
and
their
means
to
serving
customers.
A
An
exceptional
listening
experience
in
our
case
and
really
for
us
they're
there
a
way
that
we
can
kind
of
collectively
work
with
this
community
to
build
really
sound
underpinnings
to
move
developers
faster
and
faster
in
in
their
quest
to
build
features
and
incredible
experiences.
But
but
there
was
a
time
before
them
and
in
it
really
it
wasn't
always
this
way
and
and
frankly,
I
wasn't
even
at
Spotify
for
a
lot
of
some
of
the
history
that
we'll
share
with
you
and
I'll
hand
it
over
to
Dave
to
do
a
little
bit
of
a
background
and.
D
I
wasn't
around
for
most
of
my
history
either,
but
I'll
still
just
talk
about
it
anyway.
So
I
think
a
few
of
our
talk
to
talk
about
some
of
this,
but
it's
really
important
kind
of
setting
the
stage
for
it
someone's
not
we're
talking
about
so
it
starts
when
you
were
still
in
data.
Centers
I
had
a
time
before
all
those
fun
buzzwords.
D
So
we
weren't
terribly
worried
about
things
like
whether
or
not
the
way
our
service
was
interacting
with
each
other
is
open
source
or
powered
by
community
or
we'll
work
tomorrow,
because
we
just
really
needed
it
to
work
today
and
we'll
deal
with
tomorrow.
Tomorrow
and
at
that
point
we
solved
a
lot
of
really
interesting,
really
challenging
engineering
problems
that
we
often
spoke
about
at
conferences
like
this,
but
we
solved
them
very
much
in
our
own
proprietary
ways.
D
Then
we
got
to
a
point
where
we
realized
that
we
really
didn't
want
to
be
in
the
business
of
managing
any
of
our
own
Hardware.
We
had
a
lot
of
the
kind
of
typical
reasons
for
wanting
to
move
to
cloud
that
everyone
talks
about.
They
don't
want
to
manage
their
hardware.
They
don't
want
to
manage
their
switches.
They
don't
want
to
have
to
worry
about
various
things
like
a
network,
cable
dies
in
a
data
center
somewhere,
and
you
have
to
send
someone
out
and
fix
it
all
of
that
fun
stuff.
D
So
we
decided
to
move
to
public
cloud,
and
at
this
point
we
really
were
focusing
entirely
on
those
couple
things
I
said,
so
we
wanted
to
fork
lift
into
cloud.
Some
people
called
it
lifting,
so
we
were
running
effectively
the
exact
same
architecture.
We
are
on
Prem,
just
in
the
cloud
we
were
instead
of
running
on
PM's
on
our
own
Hardware
running
at
the
amp.
D
We
didn't
want
to
focus
on
multiple
things
at
once,
and
that
worked
extremely
well
for
this
migration
and,
on
the
other
hand,
it
kind
of
put
us
in
a
place
where
there
was
a
clear
next
step,
a
clear
next
migration,
and
even
while
we
were
doing
that
first
one.
We
had
been
talking
a
lot
about
cloud
native
and
at
that
point
we
just
used
the
term
and
we
hadn't
really
defined
what
it
meant
to
a
lot
of
people.
D
D
We
just
said
whenever
we're
done
with
this
thing
will
be,
will
work
on
the
cloud
native
thing
and
that's
kind
of
where
we're
now
we
recently
moved
on
from
shutting
down
our
data
centers
and
started
working
on
this
and
we've
started
by
trying
to
define
cloud
native
and
then
work
towards
it,
and
here
just
like
last
time,
we
have
kind
of
a
core
set
of
goals
and
a
couple
of
the
obvious
ones
are
things
like
resilience.
A
lot
of
their
resilience
patterns
in
cloud
are
very
different
from
we've
seen
our
data
centers.
D
We
on
the
one
hand,
see
many
more
failures
and,
on
the
other
hand,
the
failures
are
much
much
shorter,
live
than
much
quicker
to
recover.
So
all
of
the
things
that
we
used
to
do
to
recover
from
failures
in
our
data
centers
don't
really
work
as
well
anymore.
So
we
have
to
change
a
lot
of
those
practices
to
truly
take
advantage
of
the
cloud.
D
Obviously,
operability
things
like
not
having
to
worry
about
the
operational
toil
of
dealing
with
all
of
our
data
centers
with
much
less
and
then
there's
a
bunch
of
easy
wins
that
I'm
sure
you've
all
heard
of
with
cloud
things
like
quick
provisioning
and
all
of
that
I
mean
there's
a
bunch
of
observability
like
both
wins
and
challenges.
Obvious
challenges
like
in
a
very
ephemeral
environment
like
cloud
are
existing
on
Prem,
not
very
ephemeral,
built
tools.
D
Don't
work
very
well,
then
there's
a
bunch
of
next-level
things
like
looking
at
how
to
be
more
cost-effective
in
cloud
where
we
can
do
things
like
get
different
machines.
I.
Don't
think
like
that,
but
the
overall
goal,
much
the
same
as
it
was
when
he
moved
to
cloud.
His
focus
is
allowing
us,
as
a
music
streaming
company,
to
focus
on
many
of
our
people
as
many
of
our
financial
resources
on
being
a
better
music
streaming
company,
as
opposed
to
a
better
infrastructure
company
or
a
better
VMs
or
hosting
company,
or
any
of
these
stages.
D
So,
like
I
said,
this
really
is
kind
of
two
separate
jumps,
one
to
cloud
one
to
cloud
native
and
I
kind
of
asked,
which
is
harder,
but
I
think
I've
already
really
set
up
that
clearly.
The
second
one
is
the
much
harder
one,
because
the
way
that
we
did
the
first
one
we
kind
of
cut
a
line
of
where
the
layer
abstraction
is-
and
it's
really
a
kind
of
very
physical
infrastructure
and
we're
moving
out
of
that.
Whereas
all
of
the
next
step
is
at
a
much
higher
layer,
it's
much
closer
to
people's
business
logic.
D
A
lot
of
it
actually
is
architecture
of
their
services
to
make
them
truly
work
in
its
cloud
native
environment
and
I
want
to
touch
a
little
bit
on
some
of
the
changes
that
we
have
to
make
across
our
architecture
across
some
of
our
services
to
get
there
and
I
think
I
just
said
had
to
make,
but
we're
very
much
still
in
the
process
of
doing
it.
So
I
can't
use
the
past
tense
there.
So
let's
go
back,
let's
go
way
back,
but
not
quite
to
1985,
probably
a
little
closer
2010.
D
We
built
all
of
these
things
to
operate
all
of
the
core
infrastructure
across
Spotify
and
there
were
a
bunch
of
obvious
things
we
needed
like.
We
needed
a
really
good
inner
service
communication
framework
and
we
started
by
using
simple
things
that
were
readily
available
like
HTTP,
1
and
rest,
and
we
found
a
bunch
of
reasons
why
that
didn't
work
at
very
high
scale,
especially
for
large
amounts
of
binary
data
that
we
were
passing
around
for
various
things.
So
we
needed
something
and
at
that
time
we
didn't
have
any
great
alternatives.
D
So,
after
not
finding
anything
easily,
we
built
our
own
thing.
We
built
this
thing
called
Hermes
on
0
mq,
same
thing
in
the
container
registration
world.
We
at
that
point
had
transitioned
or
were
working
on
transitioning
all
of
our
back-end
services
to
run
in
docker
containers,
and
we
need
something
that
could
orchestrate
the
number
of
services
we
had
across
all
of
our
BMS
reliably
and
in
2013
there
wasn't
really
a
good
option
out
there,
having
not
found
one
we
built
born.
D
We
built
Helios
the
thing
that
Dan
Cohen
mentioned
the
other
day
same
thing
for
monitoring.
We
needed
a
scalable
monitoring
solution
that
could
monitor
all
of
our
services
and
do
both
storing
all
that
data
in
the
time
series
database,
as
well
as
giving
graphing
and
dashboarding
solutions
to
all
of
our
teams
in
a
good
multi-tenant,
easily
managed
way.
We
didn't
find
anything
great,
so
we
built
our
own
things,
some
based
on
open
source
solutions,
some,
not
we
open
sourced
the
time
series
database.
D
We
didn't
open
sourced
through
correctly
the
reporting
tool,
but
we
still
built
all
of
it
and
because
we
had
all
these
custom
things,
there
were
a
bunch
of
kind
of
next-level
things
like
tracing
that
we
just
didn't
have
because
there's
no
easy
way
to
get
tracing
kind
of
for
free
on
a
completely
custom.
You
know
inner
service
communication
platform
like
ours.
D
We
open
sourced
a
bunch
of
things
ranging
from
maven
plugins
to
data
orchestration
tools,
and
some
of
them
did
very
well
actually
like
the
date
or
creation
tools
around
Luigi
and
things
like
that
and
our
maven
blocker
plugin
today.
Well,
but
on
the
other
hand,
we
open
sourced
hundreds
of
other
things
and
most
of
those
really
are
still
only
used
by
us
and
the
same
thing
kind
of
goes.
The
other
way.
As
we
looked
around
trying
to
find
whatever
solutions
we
could,
if
something
open-source
existed,
that
worked
for
our
solution
were
youth
for
our
problem.
A
So,
to
summarize
here
to
summarize
here,
like
the
really
we
we
tried
to
open
for
sweet,
we
did
open
source
in
a
lot
of
ways,
but
but
it
wasn't
a
big
focus
and
it
wasn't
something
that
we
could
fully
invest
in
and
jump
jump
in
on
in,
and
so
these
tools
didn't
come
down
side-effects.
We
had
a
bunch
of
positive
ones,
we're
still
here.
The
company
exists,
we
build
things
they
work
and
and
that
infrastructure
that
we
build
it's
scaled
and
most
important.
A
It
gave
us
this
underlying
confidence
that
we
could
solve
challenge
and
challenging
engineering
problems,
and
in
do
so
in
a
fast,
effective
and
scalable
way,
but
it
came
with
some
less
positive
ones
too.
We
we
had
this
occasional
divergence
from
the
industry
and
in
particular,
now
CN
CF
in
then.
That
makes
it
hard
to
compose
these
technologies
over
time
as
new
ones
come
and
you're
kind
of
relying
on
on
proprietary
or
in-house
ones.
A
It's
hard
to
take
the
assumptions
of
the
new
systems
and
compose
them
together
nicely,
and,
secondly,
we
had
this
bias
toward
building
versus,
adopting
or
collectively
building
to
solve
these
cases.
They've
talked
about
it,
and
this
is
really
the
the
more
challenging
one
this.
This
is
the
one
that
that
I
think
we
have
struggled
with
still
because,
because
the
the
first
one
it
it's
it's
it's
okay,
and
then
we
can
figure
it
out
and
what
we
hear
is
like
we're
not
alone
in
these
there's
a
lot
of
people
in
this
position.
A
A
The
case
study
on
G
RPC
and
essentially
the
tying
back
to
the
title.
It's
you
we
say
like
oops.
We
did
it
again.
It's
we're
referring
to
those
those
technologies
that
we've
built.
Where
we're
now
there
are
or
future
there
will
be
industry
standards
in
those
spaces
and
figuring
out
how
we
don't
go,
build
a
gr
PC
and
instead
how
we
build
one
thing,
collectively
and
kind
of
skate
to
where
everybody's
going
so
gr,
PC
likely.
If
you're
here
you'd
know
about
it,
you
know
what
it
is.
It
was
established
in
2015
ish.
A
Obviously,
Google
had
things
before
that.
Stubby
and
so
it's
an
open
model,
a
modern
open-source,
RPC
framework,
lots
of
you
use
it.
It's
maintained,
documented
and
supported
by
everyone
for
everyone.
It
can
be
run
anywhere.
Ok,
it's
a
little
hard
on
Web
Apps
in
those
smart
devices
we
mentioned,
but
we're
helping
trying
to
figure
that
out
and
it
supports
awesome
things,
load,
balancing
tracing
health,
checking,
authentication
and
more
and
those
things
are
all
supported
out
of
the
box.
A
They
aren't
things
that
you
have
to
go,
build
and
support
for
a
use
case
that
you
need
it's
it's
kind
of
already
built
in
and
provided
by
by
the
technology
itself,
then
introducing
as
Dave
mentioned
Hermes,
and
no
it's
not
the
the
Greek
god
of
trade
heralding
merchants,
commerce,
roads,
thieves,
trickery,
sports,
travelers
and
athletes
from
Greek
mythology.
Nor
is
it
the
Parisian
brand
with
trendy
scarves
and
those
hideous
Apple
watches
it's
it's
an
in-house
RPC
ish
framework,
and
it's
something
that
we
introduced
in
2012
fish,
it's
internal,
so
Spotify!
It's
not
something!
A
That's
been
open-source.
It
supports
our
existing
use
cases.
It's
well
maintained.
It's
well
documented.
It's
well
tested!
It's
used
throughout
the
Spotify
stack
Spotify
engineers
have
good
familiarity
with
it
and
frankly
it
works.
It's
not.
It's
not
a
really
a
big
end
point,
and
so
the
question
is
like:
why
would
we
transition
I
mean-
and
this
was
something
that
we
were
met
with
as
maintainer
xand
builders
of
infrastructure?
It
was
like.
Well,
we
want
to
move
so
that
we
can
compose
these
things
and
we
kind
of
came
up
with
this.
A
This
set
of
things
that
we
thought
was
valuable.
We
thought
these
were
wins.
We
thought
these
were
the
ticket
to
selling
people
on
ER
BC
at
Spotify,
there's
less
boilerplate.
We
can
generate
clients,
that's
awesome,
but
engineers,
a
lover
right,
it'll
be
quicker,
onboarding,
not
just
like
new
languages,
but
new
developers
as
well.
A
new
developer
will
come
in
from
from
another
company
or
another
role
and
they'll
know
what
GRC
is
and
they'll
know
how
to
use
it,
and
perhaps
they've
used
it
in
past
roles
and
and
that's
kind
of
a
take
in
understanding.
A
Instead
of
figuring
out
that
Hermes
exists
what
it
is
and
how
we
use
it
and
that
it's
actually
not
that
confusing.
It's
easier
integration
with
third-party
tooling
envoy
service
mesh,
we're
at
a
speed
shop.
So
a
lot
of
the
Google
cloud
technologies,
sort
of
take
it
as
an
assumption
in
some
places,
or
at
least
definitely
support
it,
and
then
you
certainly
have
the
resiliency
patterns
circuit,
breaking
hedging
deadlines.
Things
like
that
that
it's
known
for,
and
so
we
we,
we
kind
of,
came
out
and
we're
like
these
are
the
upsides
right
and.
D
We
quickly
saw
that
a
lot
of
people
didn't
really
care,
because
the
people
that
a
lot
of
you
Lawson
just
talked
about
were
new
engineers.
Onboarding
are
new
things,
onboarding
and
the
people
have
to
migrate.
The
GRP
see
are
the
current
engineers
on
their
current
services
and
with
for
them,
looking
at
a
bunch
of
the
things
that
we
just
threw
on
that
list
didn't
really
matter
because
they
already
have
their
existing
resiliency
patterns
and
the
way
they
do
things.
D
That's
a
lot
of
work
to
rebuild
all
of
their
service,
endpoints
and
potentially
their
clients
for
what
felt
like
very
little
gain
to
them,
and
they
were
completely
right
because,
like
Austin
said
what
we
had
worked
pretty
well
like
Hermes
worked
very
well
for
all
the
use
cases
we
had
and
we
were
speaking
to
them
in
about
things
we
cared
about,
not
things
they
cared
about
and
I
took
this
picture
last
year
at
cube.
Con
I
forget
who
was
actually
doing
the
presentation,
so
I
can't
give
him
credit
but
effectively
found
the
same
problem.
D
So
we
both
kind
of,
should
have
known
this
already
and
should
have
talked
to
some
people
and
know
this
already,
but
most
developers
really
just
want
to
build
something
and
ship
it
quickly.
They
don't
want
have
to
worry
about
whatever
the
underlying
infrastructure
is
or
like
this
thing
says,
whatever
their
timeout
values
are
or
all
of
these
things
like
what
port
their
services
are
on,
they
just
want
their
stuff
to
work
in
the
introductory
team
to
handle
it.
D
We
started
using
this
metaphor:
like
won
a
few
of
us
of
it
being
just
actual
infrastructure
like
when
you're
driving
your
car.
You
want
to
get
from
point
A
to
point
B.
You
don't
really
want
to
know
about
the
infrastructure
problems,
that's
exit,
ramp
or
some
crazy
highway.
Has
you
just
wanna
be
able
to
drive
on
it
or
be
redirected
around
it
and
get
there,
but
not
be
told?
Oh,
these
are
the
details
of
it.
D
Here's
why
you
have
to
go
there
and
do
these
30
things
to
build
an
exit
to
go
around
it,
and
so
all
that
stuff
is
great.
But
it's
really
not
the
thing
that
people
truly
cared
about,
and
then
we
started
digging
in
with
some
of
the
people,
because
a
bunch
of
the
same
people
that
wrote
those
Stack
Overflow
posts,
saying
if
they
didn't
see
much
value
and
the
things
we're
asking
for
we're,
also
very
strong
evangelists
for
G
RPC.
D
So
we
have
to
figure
out
well,
if
you
are
so
against,
learn
if
you
don't
see
how
much
value
in
the
things
that
we're
pushing.
Why
are
you
so
excited
about
this
and
they
come
up
with
a
whole
bunch
of
other
really
interesting
things.
The
first
obvious
few
are
things
like
message:
integrity,
it's
a
lot
harder
to
build
encryption
and
really
good
message:
integrity
of
things
like
even
just
kind
of
simple
checks,
any
mechanisms
all
the
way
across
all
of
our
services
when
all
of
our
code
for
inter-service
communication
is
proprietary
in
our
own.
D
Of
course,
we
can
do
it
and
then
roll
it
out
across
the
fleet
and
get
everyone
running
there,
but
it
takes
a
lot
of
effort
and
something
like
using
G
RPC
and
using
a
bunch
of
built-in
features
that
we
can
get
from
our
cloud
provider.
We
just
give
us
a
lot
of
that,
so
it
felt
like
an
obvious
thing
that
they
really
want,
but
the
other
thing
is
G.
Rpc
becomes
more
and
more
widely
adopted
and,
like
Austin
said
we
are
enjoy
where
you
are
in
GCP
and
a
lot
of
GCP
services.
D
We
interact
a
lot
of
GP
services
over
T
RPC
anyway,
so
if
all
of
our
interactions
with
those
that
platform
is
over
chair
PC,
then
a
lot
of
the
practices
we
might
do,
whether
it's
for
the
message,
integrity,
things
or
even
just
for
authentication
and
and
of
every
other
layer
of
you
know,
service
communication
becomes
the
same
across
the
board
and
then
things
become
much
much
easier
for
those
same
developers
that
might
not
care
about
easier,
onboarding
or
less
boilerplate.
Now
they
care,
because
everything
is
uniform.
D
Everything
is
the
same
and
the
other
really
big
win
in
that
uniformity
space.
It's
global
monitoring
when
everything
is
running
on
the
same
platform
when
everything
is
running,
G,
RPC
and
every
service,
as
well
as
every
cloud
provider
provided
service
and
most
everything
interact
with,
has
the
same
set
of
metrics
and
you
easily
can
see
all
the
things
you
care
about,
whether
it's
latency
or
error
rates
or
request
rates,
anything
about
in
the
exact
same
way
and
the
same
types
of
dashboards
in
the
same
places,
and
that
was
actually
really
valuable.
A
Into
the
things
that
we
learned
in
this
process
and
as
we
continue
on,
but
with
other
use
cases
and
try
to
as
for
the
experience,
so
it's
not
not
dependent.
We
we've
learned
a
couple
of
key
things
and
in
kind
of
three
takeaways
that
we
wanted
to
leave
you
it.
The
first
is
that
infrastructure
problems
are
not
developer
problems,
largely
when
you're
in
this
infrastructure
domain
or
in
this
platform
domain
or
whatever
you
call
it
within.
Within
your
context.
A
There
they're
not
the
same,
we're
trying
to
build
things
that
accelerate
and
enable
and
make
developers
contributing
to
these
customer
experiences,
sometimes
just
the
same
developer
in
a
in
a
different
part
of
their
job.
Ten
times
faster.
You
can
actually
slow
them
down
by
getting
into
the
details
of
a
lot
of
what
these
technologies
are,
and
it's
it's
up
to
these
infrastructure
teams
to
figure
out
ways
as
we
build
platforms
on
platforms.
A
This
has
been
a
common
theme
to
make
sure
that
we're
focusing
on
who
the
right
customer
is
in
that
case,
and
in
this
case
that
customer
is
these,
these
developers
or
people
building
features
that
actually
provide
value
to
the
the
end
customer
your
products
actively
kind
of
calling
out
this.
This
sentiment
of
of
things
like
not
being
invented
here
is
actually
something
that
we
found
hugely
important.
A
We
we
saw
that
in
the
in
the
process
of
kind
of
shifting
the
conversation
for
more
than
gee,
our
PC
is
really
good
and
Hermes
sucks,
like
people
were
still
sort
of
opposed,
even
when
we
presented
it
in
terms
of
the
problems
that
it
solved
for
them
in
the
and
and
at
that
we
sort
of
realized
that
this
this
not
invented
here,
something
you've
heard
it
probably
other
Spotify
talks
this
week
or
another
conferences
from
us.
Is
this
big.
A
Actually
the
community
can
help
us
learn
and
move
a
lot
faster.
We
can
learn
from
the
mistakes
of
others
again.
That's
the
intention
of
this
talk
is
to
share
some
of
our
mistakes
with
you,
so
you
can
learn
how
to
move
faster,
but
also
technologically,
we
can.
We
can
iterate
quickly.
We
could.
We
can
pull
together
our
collective
experience
and
design
and
build
better
products
and
I
think
you're,
seeing
that
with
kubernetes
project,
where
sometimes
it
can
feel
like
the
latency
of
getting
a
proposal
in
and
then
the
code
in
emerged.
A
It
seems
like
it's
on
a
meteor,
a
lot
of
like
just
a
crazy
time
scale,
but
but
really
what
that
is,
is
it's
all
of
these
different
experiences
and
of
the
community
coming
together
to
make
sure
that
we're
building
the
right
thing
and
it's
well
informed,
and
so
with
that
we're?
Where
are
we
in?
What
are
we
actually
doing
with
GRP
see?
This
is,
of
course,
our
favorite
geo
PC
logo,
with
with
pancakes
having
been
announced
at
the
last
G
or
PC
conference.
A
We're
we're
focusing
a
lot
on
on
the
the
tooling
for
clients,
as
I
sort
of
made
the
dig
earlier
like
what
is
G
RPC
on
web.
Look
like
it's
it's.
Obviously
the
project
is
out
there.
You'll
see
us
at
this
point.
We've
certainly
created
issues
for
it
and
have
done
a
ton
of
work
to
actually
see
how
it
could
be
integrated
for
many
of
the
web.
Applications
that
we
do
have
mobile
on
clients
is
something
that
we've
had
a
lot
of
challenge
with.
A
It's
a
it's
a
big
challenge
and
latency
and
performance
are
hugely
top
of
mind
there,
especially
for
us
and
so
figuring
out
what
the
right
tools
are
for
that
end
and
how
it
works
is
key.
Then,
just
G,
RPC
tooling
in
general,
there's
there's
tons
of
tools
that
we'd
like
to
build
and
developers
internally
are
either
working
to
build
and
contribute
back
or
buying
for
the
time
to
build,
and
so
our
hope
is
to
build
this
infrastructure
together
and
I.
Think
that's
a
common
trend.
A
You'll
hear
from
Spotify
is
this:
how
can
we
build
infrastructure
together?
We
say
it
within
spotify
all
the
time,
but
also
we
say
it
to
you.
Is
the
community?
How
can
we
do
this
and
how
can
we
kind
of
all
join
hands
and
do
it
together
and
those
are
three
places
we
see
a
ton
of
leverage
I'm
an
opportunity
and
then
internally
a
big
success
story.
We've
seen
is
actually
just
creating
a
gr.
Pc
users
community
has
been
hugely
effective.
A
We
it
started
with
a
bunch
of
people
coming
in
and
actually
asking
questions
in
in
the
infrastructure.
Folks,
you
know
sharing
their
insights
or
tools,
people
could
use,
but
now
we
found
that
the
infrastructure
folks
hardly
answer
any
of
the
questions
anymore.
It's
all
people
within
the
community
who
have
integrated
it
in
other
parts
of
the
company
for
completely
different
use
cases
I'm,
actually
sharing
their
insights
into
to
wrap
it
up.
A
You
know
we,
we
hope
our
learnings
can
help
you
and-
and
we
would
love
to
kind
of
get
feedback
on
any
of
these
we'd
love
to
get
feedback
on
kind
of
the
focus
that
we're
taking
with
JIRA
PC
and
where
we
feel
like
we
can
optimize
and
we
want
to
do
it
together.
Our
hope
is
to
build
this
infrastructure
and
in
other
together
as
we
move
forward,
and
with
that
you
are
welcome
to
ask
us
questions
now
in
this
forum.
A
B
How
are
you
storing
them
and
are
you
compiling
you
know
a
big
package
that
includes
all
clients
and
all
servers,
and
how
do
you
manage
all
that
because
we
have,
we
have
been
net,
we've
been
doing
the
same
thing
as
you
have
been
doing,
and
we
ended
up
with
a
single
people
for
all
Protoss,
but
we
are
still
not
we're
still,
not
sure
if
that
was
the
right
decision.
So
what
are
you?
What
are
you
take
on
on
that?
One.
A
You
can
take
it.
I
mean
I'll
share
that
we
started
with
a
mistake.
We
started
by
building
tooling
to
go
support
this
and
kind
of
taking
this
like
ooh,
that's
a
big
challenging
problem.
Let's
go
build
a
tool
to
solve
it
and
you
can
share
kind
of
where
we
are
now
and
what
we're
actually
doing
once
we
realized
that
building
a
new
machine
was
not
the
answer.
Yeah.
D
Yeah,
so
our
initial
idea
was
to
build
not
just
a
central
repo
but
a
full
set
up
like
a
bunch
of
tools
around
it,
like
a
nice
webpage
that
looks
like
swagger
all
this
stuff
that
generates
all
your
clients,
does
kind
of
everything
you'll
ever
want,
and
we
quickly
realized
that
we're
effectively
building
an
entirely
new
package
manager.
For
all
of
these
things
and
it's
a
ton
of
work
and
at
that
point
we
had
a
small
handful
of
JRPG
services.
D
So
it
would
be
a
ton
of
work
for
like
five
or
six
teams
that
may
or
may
not
even
necessarily
care.
So
right
now
what
we
have
is
kind
of
like
what
you
said.
We
did
have
a
central
repo
of
all
the
proto's
and
we've
built
a
few
simple
tools
for
doing
things
like
pulling
proto's
from
there
and
generating
clients
like
a
maven
plug-in
that
can
pull
from
that
repo
and
generate
a
Java
client
or
the
Java
server
stubs.
D
I
think
Matthias
mentioned
ubers
proto-tool
like
we're
starting
to
look
at
a
bunch
of
other
thing,
is
to
pull
them
in,
but
I
can't
say
we
have
like
the
perfect
answer
yet
because
we're
not
at
a
point
of
enough
adoption
to
say
that
we
have
something
that
everyone's
happy
with,
but
the
shared
repo
thing
seems
to
be
working
pretty
well
so
far.
So.
C
D
Look
she
was
saying
that
are
the
reasons
of
the
information
was
really
interesting.
Gfp
C
ended
up
being
different
than
the
reasons
that
developers
across
the
company
cared
about
geo
PC.
So
how
do
we
reconcile
that?
The
next
time
we
want
to
migrate?
Something
or
the
intriguer
team
decides
that
something's
right?
How
do
we
make
that
work
for
developers
across
the
company
yeah.
A
A
This
is
not
it's
it's
not
in
terms
of
infrastructure,
management
and
maintenance
value,
but
it's
in
terms
of
actual
end,
developer
value
and
I.
Think
it's
similar
to
what
you
hear
a
lot
of
like
smaller
companies,
preaching
like
somebody
like
Airbnb,
coming
out
and
saying
that
it's
it's
all
about
the
experience
for
like
that.
One
end
customer
like
let's
sell
for
one
customer
and
so
for
us.
A
It's
the
same
thing
and
just
applying
that
to
just
this
infrastructure
domain
and
it
extends
to
even
how
we
market
things
like
this,
like
in
the
G
RPC
case,
changing
kind
of
the
angle,
but
even
like
creating
visuals.
Creating
the
Spotify
stickers
like
getting
people
excited
about
those
sorts
of
things
and
making
sure
that
that
value
proposition
is
clear.
It
has
been
a
huge
investment.