►
Description
Presented by Kelsey Gilmore-Innis, Engineering Manager (GitHub).
We present the story of Moda—GitHub's internal service platform. Moda is a set of tools, practices, and ideas for running software on Kubernetes. We’ll share how we approached big infrastructure changes, what has worked well, and why we made the decisions we did. Our success is rooted in treating infrastructure like a product, remembering that we build tools to enable other developers, and focusing on what can help them build the future of software.
This talk is about the big picture, not the day-to-day details, so you won’t need prior knowledge of containers or Kubernetes to participate or benefit.
About GitHub Universe:
GitHub Universe is a two-day conference dedicated to the creativity and curiosity of the largest software community in the world. Sessions cover topics from team culture to open source software across industries and technologies.
For more information on GitHub Universe, check the website:
https://githubuniverse.com
A
Hi
everyone
welcome
hello.
You
are
at
infrastructure
as
product
building
get
hubs
feature
and
in
this
session
we're
going
to
talk
about
that
very
idea.
Treating
your
internal
infrastructure
as
a
product,
I'm
gonna
explain.
You
know
what
that
means
to
me.
Why
I
think
it's
a
good
idea
and
then
we're
gonna
cover
some
of
the
important
themes
of
that
practice
and
then
illustrate
those
themes
with
examples
from
github
x'
work.
A
To
be
clear.
This
talk
is
not
a
rundown
of
github
x'
infrastructure
strap
stock.
Nor
is
it
a
tutorial
about
kubernetes.
Some
of
those
tools
will
come
up,
but
we're
gonna
focus
on
how
infrastructure
impacts
people
rather
than
any
technical
details,
so
don't
leave
yet
so
before
we
get
too
far
in
I'd
like
to
introduce
myself
first
things.
First,
I
am
NOT.
Sadly,
Sophie
Haskins,
I'm,
Kelsey,
Gilmore,
NS
I'm,
lucky
enough
to
be
Sophie's
manager,
I
work
with
Sophie
on
an
amazing
team.
Here
at
github
called
production
engineering
delivery.
A
We
are
responsible
for
the
infrastructure
that
engineers
at
github,
used
to
run
their
code
in
production.
Sophie
wrote
these
slides
in
these
content.
She
unfortunately
couldn't
travel
to
SF
today.
So
I'm
filling
in
I
work
with
her
on
this
project
and
I
share
her
philosophy
and
her
excitement,
but
these
are
her
words
and
ideas,
so
all
credit
to
Sophie
Sophie
wants
you
to
know
that
her
career
has
mostly
been
building
infrastructure
for
internal
use
at
medium
to
large
tech
companies
whose
main
product
is
a
website.
A
Her
feelings
about
infrastructure
are
colored
by
that
experience,
and
the
way
that
we
discuss
infrastructure
today
is
gonna,
be
an
that
frame
of
reference
of
a
tech
company
who
delivers
their
product
via
a
website
or
API
so
infrastructure
as
a
product.
What
does
that
mean?
Well,
let's
get
into
it.
Let's
start
with
what
is
infrastructure
so
me,
and
my
team
work
on
infrastructure.
Specifically,
we
work
on
software
infrastructure
we
build
and
maintain
the
tools,
the
technologies
and
the
practices
that
other
developers
at
github
used
to
produce.
A
Github
itself
to
me,
infrastructure
is
made
up
of
three
things.
It's
as
I
said
tools,
technologies
and
practices.
Tools
are
things
like
configure
configuration
management
systems,
cloud
providers,
deployment,
automation,
monitoring
systems
and
source
code
repositories.
Technologies
are
things
like
Linux
servers,
containers,
everybody
loves,
containers,
caching,
virtualization
and
distributed
tracing
and
practices.
Are
things
like
continuous
integration,
checklists,
on-call
rotations
and
service
level
objectives?
A
How
does
that
differ
from
a
product?
Well,
traditionally,
products
are
like
the
things
that
your
organization
exists,
for
if
you're
a
business
they're,
how
you
make
money
a
product,
a
product
make
someone's
life
better
in
a
in
some
way.
It
differs
from
a
commodity
because
it
tells
a
story
about
how
your
life
is
better.
When
you
use
it,
it
doesn't
have
to
be
a
good
story
with
plot
and
a
conflict.
A
It
could
be
something
as
simple
as
like
our
compiler
is
faster,
so
you
have
more
time
to
focus
on
what
matters,
or
this
editor
has
a
colorful
theme
that
makes
you
smile.
Write.
A
product
story
reflects
its
creators,
vision
of
how
it's
going
to
improve
your
life
and
the
best
products
tell
a
compelling
and
a
coherent
story
about
that
vision
in
a
consistent
voice.
A
I'm
a
product
story
is
also
an
opportunity
to
build
trust
and
familiarity
with
its
customers,
the
easier
it
is
for
someone
to
see
how
a
product
would
improve
their
life
and
the
more
compelling
that
vision
is
the
more
likely
they
are
to
pay
money
for
it.
Finally,
a
product
provides
those
benefits
to
a
specific
audience.
That's
one
thing
that
distinguishes
a
product.
It's
built
to
fulfill
like
the
particular
wants
and
needs
of
a
specific
discrete
group
of
people.
A
A
So
how
does
the
infrastructure
fit
in
with
that
idea
right?
So
one
of
the
things
that
I
want
to
go
back
to
is
that
last
bullet
point
about
infrastructure
infrastructure
exists
to
help
other
humans.
I
worry
sometimes
that
folks,
working
in
an
infrastructure
forget
that
right
operations
teams
have
a
really
long
history
of
being
grumpy
and
being
hostile
to
their
users
and
I.
A
Think
we've
probably
all
experienced
that
infrastructure
has
this
reputation
of
somehow
being
more
technical
or
more
objective
than
other
disciplines
and
I've
worked
with
coworkers
who've
said
things
like
I,
don't
need
anyone
tell
me
to
side
on
what
to
work
on.
I
know
what's
right
to
work
on
or
our
system
is
fine.
It's
the
users
fault
right.
That
sort
of
problem
exists
between
keyboard
and
chair
attitudes.
I,
don't
like
this
infrastructure
exists
to
help
other
people.
A
If
you're
working
at
a
company,
it
exists
to
help
those
people
meet
business
goals,
as
mentioned,
we
often
an
infrastructure
focus
on
notions
of
like
technical
correctness,
objective
performance,
these
SLO
that
you
can
hit
or
not
hit,
but
those
those
motivations
for
us
to
build
infrastructure.
They
can
miss
with
the
humans
that
we're
supporting,
actually
want
and
need
in
order
to
do
their
jobs
well
and
in
order
to
move
the
business
forward.
A
When
we
ignore
that
human
element,
we
can
miss
some
of
the
ways
of
having
a
coherent
narrative
about
our
infrastructure
can
really
improve
it
as
humans.
One
of
the
things
that
we
do
is
we
establish
norms
through
narratives
that
we
tell
tooling,
and
technology
is
obviously
important
right,
but
infrastructure
and
practice
relies
really
heavily
on
shared
norms.
Shared
values
to
operate
some
example
of
an
of
norms
are
something
like
you
don't
deploy
without
testing
your
code.
That's
a
norm.
Right
services
should
be
thoroughly
monitored.
A
That's
another
norm,
yeah
monitoring
the
DevOps
movement,
which
I'm
sure
a
lot
of
folks
are
familiar
with,
provides
a
great
example
of
how
a
narrative,
and
that
narrative,
roughly,
is
what,
if
developers
and
operators
acted
like
they're
on
the
same
team.
It's
a
really
good
example
of
how
that
narrative
makes
for
stronger
infrastructure
systems
and
then,
finally,
to
tie
it
back
to
sort
of
the
last
definition
of
a
product.
A
Even
though
we
are
building
internal
tools
for
internal
users,
we
argue
that
we
actually
do
have
a
market
somewhat
analogous
to
what
xrx
no
products
do
we
create
our
internal
infrastructure
for
a
specific
set
of
users
right,
it's
our
co-workers,
specifically,
it's
our
developer,
coworkers
and
it's
actually
even
often
a
smaller
section
of
those
and
those
those
people
have
a
specific
set
of
pain
points.
Now,
in
some
cases,
if
we
don't
serve
those
problems
well,
we
can
actually
lose
those
customers
right.
A
They
may
be
able
to
choose
a
third
party
infrastructure
provider
and
infrastructure
product
like
a
juror
or
Heroku,
but
even
if
you
do
have
a
mandate
to
provide
infrastructure
to
your
org
because
of
how
important
those
norms
are,
adoption
of
an
engagement
with
your
infrastructure
is
really
critical
to
its
success
and
anyone
who's
worked
in
operations
or
infrastructure
will
tell
you
right.
Devs
will
go
around
you.
They
will
do
the
bare
minimum
to
get
you
off
their
back
and
they
will
resist
you
if
your
infrastructure
doesn't
meet
their
needs.
A
So
you
actually
kind
of
have
pretty
similar
motivations
to
external
products
in
terms
of
wanting
to
find
and
create
that
fit
between
your
product
and
your
market.
So
why
why
now?
Why
is
this
something
we
decided
to
focus
on
now
or
that
we
think
is
important
now
couple
reasons:
one
is
the
DevOps
movement
which
I
mentioned
you
know
the
DevOps
movement
has
sent
waves
to
the
industry
and
it's
really
raised
the
bar
for
infrastructure
engineers.
It's
asked
us
to
think
beyond
just
the
technical
right
and
to
think
about
humans.
A
Another
reason
that
we
think
it's
a
good
idea
now
is
that
every
year,
more
and
more
tools
come
out
that
help
extract
away
more
and
more
of
that
sort
of
busy
work
that
toil
of
boilerplate
operations,
the
less
that
we
have
to
focus
on
getting
the
machine
to
work.
The
more
time
we
have
to
focus
on.
Why
does
the
machine
work
this
way
and
critically?
What
is
this
machine
for
very
important
question?
A
The
kubernetes
project,
great
great
example
of
this
right,
the
abstractions
that
kubernetes
uses
are
really
well
suited
to
the
problem
domain
of
running
some
software
on
some
serve,
which
is
a
problem
that
lots
of
us
are
familiar
with.
It's
a
it's,
a
rich
toolkit
for
doing
that,
and
doing
it
reliably
it's
a
basis
that
allows
us
to
ask
more
about
what
are
we
running
and
why,
instead
of
how
do
I
get
it
to
run?
A
That
means
we
might
not
be
able
to
where
you
might
increasingly
be
less
able
to
do
well
to
just
guess.
What's
right
to
build,
we
need
to
focus
how
we
build
our
infrastructure
on
humans
rather
than
on
computers.
All
right.
How
do
we
do
that?
Well,
I'm
glad
you
asked
let's
talk
about
what
does
an
infrastructure
internal
infrastructure
product
mean?
We
have
a
couple
themes
that
we
think
are
important.
A
We're
going
back
to
this
idea
that
a
product
has
a
coherent
narrative
and
a
clear
audience,
and
so
the
three
things
we
think
you
need
to
do
this.
Well,
our
vision,
resonance
and
trustworthiness
vision.
So
building
good
products
requires
vision.
It
requires
you
being
able
to
see
possibilities
to
have
impact
beyond
what's
immediately
visible
working.
A
An
infrastructure
is
actually
a
really
good
place
to
do
this,
because
it
gives
you
a
unique
window
to
technical
possibilities,
company
goals
and
social
dynamics
you're
in
a
position
in
infrastructure
to
see
a
future,
that's
better
than
just
what's
over
the
next
hill.
Just
what's
beyond
the
next
outage.
Right
and
vision
is
by
no
means
unique
to
a
product
approach
and
actually,
in
my
experience,
infrastructure
engineers
have
really
cool
visions
of
what
the
future
could
be.
A
If
the
right
pieces
fell
into
place,
they
can
see
what
a
good
system
they
often
have
a
feel
of
what
a
good
system
would
look
like,
and
they
wish
they
worked
in
one,
but
a
lot
of
times.
Infrastructure
teams
have
a
more
day-to-day
tactical,
smaller
focus
and
they
aren't
given
space
or
time
to
focus
on
long-term
vision.
A
Treating
your
infrastructure
as
an
actual
product
is
one
way
to
unlock
that
vision.
It's
a
way
to
build
a
framework
for
why
you're,
taking
a
given
step
now
or
why
you're
making
a
given
trade
off
it
helps
you
organize
your
plans
in
a
way
that
management
is
more
likely
to
support
and
that
users
are
more
likely
to
engage
with,
which
feeds
back
into
that
support.
Now
this
isn't
just
a
free
ticket
to
sit
around
dreaming
all
day,
and
you
know:
building
castles
in
the
sky
being
empowered
to
execute
on
vision,
comes
with
responsibility.
A
You
have
to
be
able
to
follow
through.
You
have
a
responsibility
to
weigh
different
options
and
choose
what's
best
for
the
organization
over
a
longer
period
of
time,
so
vision
is
really
the
backbone
of
your
products.
Narrative.
It's
why
you're
making
a
product
instead
of
just
working
reactively
and
putting
out
fires
as
they
come
up
so
that
story
now
remembers
products
tell
a
story
about
how
they're
gonna
make
someone's
life
better.
If
people
are
gonna
choose
your
product,
though,
that
story
has
to
resonate
with
them
right.
Not
all
stories
are
for
all
people.
A
You
won't
improve
anyone's
life
if
they
don't
choose
to
be
part
of
that
story
and
part
of
telling
a
story
that
really
resonates
with
your
users
is
telling
it
in
a
way
that
they
understand-
and
this
is
honestly
something
that
we
think
infrastructure
engineers
ourselves
included
struggle
with.
It
has
to
be
relevant
to
your
users
experience.
A
It
has
to
be
in
a
language
they're,
familiar
with
told
in
a
setting
that
they
feel
comfortable
in
your
stories
should
target
their
level
of
expertise
and
also
acknowledge
and
address
the
external
pressures
that
they
feel,
which
may
be
different
than
the
external
pressures.
You
feel,
and
it
isn't
just
about
telling
the
story
in
a
way
that
sounds
good
or
looks
pretty
not
that
kind
of
formulation
assumes
that
you're
telling
the
right
story
to
begin
with
a
vision
isn't
enough
to
make
a
resident
product.
A
You
also
need
beyond
the
vision
you
need
to
talk
to
your
users.
I
need
to
listen
to
how
they
view
their
problems.
I'm
being
condescending
is
really
kind
of
a
big
problem
with
infrastructure
and
operations
teams.
I
think
most
people
have
encountered
this.
If
your
users
think
that
you
don't
respect
them,
they're
not
going
to
trust
you,
you
have
to
be
humble,
you
have
to
be
a
listener
and
you
have
to
be
open
to
ideas
that
you
haven't
thought
about
before
or
that
maybe
even
seem
wrong
to
you
on
the
face
of
it.
A
It's
very
hard,
actually
cuz
you're
asking
them
to
invest
in
something
that
may
not
be
complete
yet
you're
asking
them
to
do
some
work
now
to
do
some
buy-in
not
to
support
what
you
know
is
awesomeness,
but
that
won't
come
till
later
and
you're
asking
them
at
the
at
the
core
of
it.
You're
asking
them
to
trust
you
now,
whether
or
not
they
will
it
depends
on
a
product.
A
Unlike
a
an
element
of
your
own
presentation
and
your
own
actions,
it
depends
on
how
trustworthy
you
are,
how
you
build
and
communicate
about
your
infrastructure,
product
impacts,
impacts.
How
trustworthy
your
team
and
your
products
are
perceived
as,
and
this
might
sound
like
old
hat
to
you
right,
especially
if
you're,
if
you're
someone
who
you
know
you're
like
I'm
super
trustworthy,
I
work
all
night
to
keep
the
site
upright,
but
it's
actually
not
just
about
say,
meeting
your
SLO
s
having
a
high
number
of
9s
having
great
uptime.
A
Those
things
are
amazing,
keep
doing
them,
but
Trust
is
not
a
binary
property.
You
don't
cross
a
certain
finish
line
and
cross
a
threshold
and
then
get
it.
It's
a
feeling.
It's
extremely
human.
It
means
that
one
of
the
best
ways
to
build
chav,
to
build
trust
and
how
humans
interact
with
your
product
is
is
its
interface
is
at
the
point
that
they're
interacting
the
interface
is
sort
of
the
look
and
feel
of
your
infrastructure.
That
includes
its
literal
code.
Interfaces
could
include
an
API
could
include
scripts
it
it
includes.
A
You
know
the
UI,
obviously,
whether
that's
a
GUI
or
whether
that's
a
command
line
or
something
else
chat.
Ops
say
it
includes
a
documentation,
a
really
critical
thing
that
the
interface
includes
is
the
interactions
you
have
with
your
users.
That's
not
necessarily
something
you
can
storyboard
right,
but
what
your,
how
your
users
are
treated
and
what
the
experience
they
have
when
they're
trying
to
get
help
in
chat,
email
or
issue
trackers
is
really
critical
to
the
interface
of
your
infrastructure
product.
Some
properties
of
of
these
interface
of
a
trustworthy
interface
and
a
trustworthy
product.
A
Are
that
it's
consistent?
Do
you
communicate
with
a
consistent
voice?
Do
you
use
the
important
terms
to
your
product
consistently
across
your
document?
Documentation
and
support,
you
know:
is
it
sometimes
server,
sometimes
nodes,
sometimes
host?
You
want
to
really
narrow
down
on
that.
Do
techniques
that
users
learn
about
one
part
of
their
product,
your
product
apply
in
other
parts.
If
there's
one
way
to
deploy,
say
a
change
to
networking.
Is
that
also
how
you
deploy
a
change
to
capacity?
A
It
needs
to
be
understandable
if
your
users
are
confused
by
you,
it's
hard
to
trust
them
right,
it's
hard
for
them
to
trust
you.
You
want
to
speak
and
work
in
their
language,
and
that
can
be
really
hard
if
you've
been
steeped
in
this
a
while
and
it's
important
to
try
to
break
out
of
that
and
find
out
where
you're
losing
folks
and
then
finally
needs
to
be
aligned.
What
are
the
quickest
ways
to
gain?
A
Trust
is
if
your
infrastructure
is
clearly
designed
for
exactly
the
use
case,
that
a
user
has
on
the
flip
side
of
that.
If
you
you
know,
if
you
need
to
do
something
and
you
come
to
the
infrastructure
team
and
what
they
tell
you
to
do
seems
really
far
off
from
what
you're
trying
to
do.
Do
you
feel
like
you
can't
trust
them
right?
They
don't
understand
what
you're,
what
you're
actually
going
for
now.
These
are
all
things
that
are
under
your
control
and
they're
separate
from
the
core
technology
right.
You
can
do
this
on
docker.
A
You
can
do
this
with
bash
scripts.
You
can
do
this
on
many
different
technologies
there
about
like
the
look
and
feel
or
the
interface
that
you
present,
irrespective
of
the
implementation.
A
cool
thing
about
Trust
is
that
over
time
it
builds
up
users
who
think
that
you're
trustworthy
are
more
willing
to
take
a
leap
of
faith
and
collaborate
with
you
to
build
your
vision,
all
right,
that's
all
pretty
abstract.
Let's
see
some
actual
examples
right
so
to
illustrate
these
themes.
We're
gonna
share
stories
from
our
rollout
of
kubernetes
at
github.
A
Kubernetes
is
a
system
for
deploying
and
managing
containerized
applications
across
clusters
of
hosts.
Don't
worry
about
it.
It's
not
super
relevant
to
the
actual
examples,
but
we
can
talk
tube
afterwards.
All
you
want.
We
ended
up
in
our
rollout
of
Kubb,
taking
something
that's
normally
just
infrastructure,
which
is
running
code
at
a
web
company
and
building
it
as
a
product,
and
this
has
been
a
gradual
process.
We
didn't
start
out
knowing
we
were
gonna.
Do
this
right.
A
It
started
with
a
vision
that
kubernetes
was
going
to
be
the
backbone
of
our
service
platform,
but
we
had
only
done
a
proof-of-concept
project
with
it
to
sort
of
validate
some
base
assumptions
and
to
sort
of
get
the
new
shiny
momentum
going
among
our
team.
We
want
to
use.
This
is
going
to
solve
some
of
our
problems.
We
had
a
choice
after
we
built
that
proof-of-concept
we
had
sort
of
a
fork
in
the
road
right.
We
could
either
dive
right
in
build
out
the
whole
service
platform
for
new
services,
all
in
kubernetes.
A
You
bet
everyone's
gonna
have
to
use
this
we're
building
this
platform
is
gonna.
Take
us
a
while,
but
that's
the
future.
Another
option
that
we
had
was
to
just
go
ahead
and
fully
port
the
workload
that
we
had
proof
of
concept
'add
to
kubernetes
in
production,
so
to
take
it
a
little
beyond
a
proof
of
concept.
The
first
option
that
full
service
platform
that
was
closer
to
our
long-term
goals,
but
we
didn't
feel
like
we
had
enough
experience
with
kubernetes
to
ask
others
to
trust
a
platform
built
on
it
right.
It's
a
little.
A
That's
a
little
bit
of
hubris.
We
decided
instead
to
go
the
second
route
and
we
ported
the
entire
github
web
and
hcp
API,
which
is
mostly
in
one
code
base
to
kubernetes.
To
build
that
confidence,
we
opted
support
the
monolith
kubernetes
as
a
way
to
build
trustworthiness
in
the
technology
and
in
the
work
that
our
team
does
choosing
to
port.
The
model
has
had
a
bunch
of
advantages
to
do
that.
First,
our
monolith
is
a
big
service.
It
requires
a
lot
of
compute
infrastructure.
A
This
allowed
us
to
become
comfortable
with
building
and
maintaining
large
kubernetes
clusters
right.
We
felt
confident
that
we
could
take
on
any
workload
at
this
point.
Aramis
also
takes
a
lot
of
traffic
and
if
there
were
gonna,
be
scaling
problems
at
github
scale,
which
is
significant.
This
project
was
going
to
shine
light
on
him
right.
We
were
gonna
find
out
before
other
folks
have
gotten
invested,
and
finally,
a
lot
of
developers
at
github.
A
The
time
worked
primarily
in
the
monolith
in
this
codebase,
because
it
comprised
so
much
of
our
business
logic,
and
so
we
had
a
lot
of
expertise.
We
could
draw
on
to
verify
things.
Many
eyeballs
make
shallow
bugs
right.
We
we
knew
that
we
were
going
to
encounter
issues
because
we
had
so
many
folks
who
were
touching
adjacent
work.
A
Not
only
to
this
project,
which
was
you
know,
a
little
bit
counterintuitive
to
take.
The
biggest
chunk
first
have
benefits
for
our
team
building
trust
in
kubernetes.
It
also
helped
us
make
our
eventual
product
more
trustworthy,
so
developers
became
used
to
deploying
their
code
to
kubernetes
every
day,
while
working
in
a
stock.
They
were
already
familiar
with,
so
we
were
sort
of
limiting
the
amount
of
change
that
we
were
introducing
when
they
did
eventually
begin
to
make
standalone
services
or
extract
bits
of
code.
They
already
have
associated
coup.
A
Developers
also
became
use
to
a
platform
built
by
our
team
powering
their
code,
focusing
on
that
existing
service
meant
that
we
focused
on
building
infrastructure,
that
conform
to
developers
existing
expectations
and
since
they
already
have
trust
in
existing
systems
right,
they
trust
that
the
website
and
the
API
will
be
up
and
that
our
teams
will
work
hard
to
keep
them
up.
We're
able
to
latch
onto
that
and
sort
of
borrow
some
trust
right.
In
the
end,
we
had
the
foundations
of
trust,
see
where
the
trustworthiness
for
us
to
build
a
service
platform.
A
On
top
of,
we
delivered
infrastructure
that
powered
a
really
really
large-scale
application
right:
lots
of
traffic,
lots
of
computers,
lots
of
deploys
our
team
and
our
technology
choices
had
credibility
now,
and
we
had
also
set
expectations
on
an
interface
front.
We
set
expectations
for
how
we
would
introduce
changes
to
underlying
infrastructure.
We
would
attempt
to
make
infrastructure
changes
that
impact
devs
workflow
as
little
as
possible.
A
So
this
went
so
well
this
initial
port
that
we,
we
deployed
a
cluster
that
was
open
to
any
apps,
not
just
the
monolith
just
to
see
you
know
what
would
people
try
what
would
they
put
on
there?
And
surprisingly,
we
actually
got
quite
a
bit
of
interest.
Multiple
teams
deployed
apps
on
kubernetes,
with
our
assistance
onto
this
cluster,
and
at
this
point
we
saw
this
happening
and
we're
sort
of
thinking
how
do
I
build
on
this?
A
If
the
next
step
in
our
journey
to
kubernetes,
four
out
of
everything
running
on
kubernetes,
makes
a
strong
impression
will
really
multiply
our
trustworthiness
right,
we'll
have
these
couple
of
wins
in
a
row
and
we'll
be
in
a
really
good
spot.
But
how
do
we
make
that
good
first
impression,
and
we
did
that
by
focusing
on
features
that
were
most
resonant
with
our
users
going
back
to
that
idea
of
resonance
of
really
listening
to
the
people
using
our
product?
A
What
we
did
is
we
actually
looked
at
the
issues
that
our
early
adopters
and
our
beta
testers
had
filed
and
the
questions
they'd
asked
in
chat
and
the
the
issues
they
had
raised
and
we
looked
for
patterns.
What
had
our
users
told
us
was
important
to
them?
What
were
their
challenges?
What
were
their
dreams?
What
were
their
pressures?
A
This
is
really.
It
was
really
exciting
because
we
had
all
that
already
available
to
us.
We
didn't
have
to
go
out
and
do
further
surveys,
because
we
had
dead
bugs
that
were
filed.
We
had
interactions
with
chat,
it's
important
to
realize
what
a
wealth
of
product
information
you're
already
sitting
on
as
an
infrastructure
team.
When
we
did
this,
the
common
thread
that
we
saw
when
we
took
this
first
look
was
getting
an
app
set
up
on
the
platform,
because
our
product
was
new.
A
Our
platform
was
new
that
had
been
the
focus
of
most
of
the
early
adopter
teams
right.
They
wanted
to
get
up
and
running
rather
than
longer-term
goals,
rather
than
you
know,
stability
or
performance
they
wanted
to
that
everyone
who
was
starting
with
us
these
early
adopters
word
onboarding
into
the
product.
You
know
it
makes
sense
right.
They
were
still
building
trust
in
us,
so
they
might
be
deploying
lower
risk
things
smaller
things
newer,
apps,
because
they
wanted
to
make
sure
this
was
something
that
would
work.
A
We
also
noticed
a
handful
of
folks
had
a
really
hard
time
getting
set
up.
There
was
still
a
lot
of
boilerplate
in
this
initial
rollout,
and
some
subtle
bits
were
really
easy
to
get
wrong
in
like
hard
to
find
ways,
and
so
we
thought
hey,
there's
a
good
opportunity
here.
Right,
onboarding
on
to
our
product
is
difficult.
Let's
fix
that
so
to
make
our
onboarding
features
really
resonate
with
our
users.
A
We
started
from
that
point
and
we
did
a
little
more
research,
non
infrastructure,
UX
user
experience,
designers
often
talk
about
user
stories
or
user
journeys
as
a
way
of
mapping
out
flows,
and
we
decided
to
give
this
a
try,
they're
the
experts
in
this
right.
So
we
wanted
to
rely
on
that.
We
had
a
lot
to
learn,
but
it
was
a
really
useful
exercise.
We
wrote
out
a
user
story
of
how
to
do
the
first
deploy
of
an
application,
and
we
first
did
this.
A
We
went
through
the
process
of
how
it
was
without
any
improvements
and
at
that
time,
right
there
were
things
we
knew
were
broken
right.
There
were
things
we
knew
weren't
right
and
they
seemed
obviously
challenging
or
wrong
to
us,
but
we
wanted
to
double-check
our
assumptions
by
comparing
those
trouble
points.
We
thought
we
saw
and
we
had
identified
to
the
actual
experience
of
our
users.
A
We
reached
out
to
developers
who
had
had
significant
pain,
including
some
who
abandoned
our
planned,
abandoned
kubernetes
halfway
through
and
we
talked
to
them
about
what
their
goals
had
been
and
what
they
had
hoped.
It
would
have
been
like,
or
looked
like
these
conversations
and
user
stories
pointed
to
a
particular
set
of
features,
adding
a
new
app
required,
deploying
our
change
to
our
deploy
tools.
A
So
it
would
know
about
your
app
kind
of
circular,
so
we
moved
that
configuration
into
the
apps
repo
under
the
users
control
to
simplify
the
process,
and
then
people
were
copy,
face,
copy
pasting,
cube
configs
and
deploy
configs
to
set
up
their
apps,
but
often
from
out
of
date
or
even
broken
examples.
And
so
we
made
it
a
quick
tool
that
would
generate
a
fully
working
HTTP
service
so
that
developers
could
focus
on
their
code
rather
than
building
the
strangling
the
service
platform.
A
But
when
they
were
ready
to
deploy,
it
was
going
on
the
service
platform.
When
we
built
those
features
out,
we
regularly
did
real-life
tests
of
making
new
test
applications
in
order
to
experience
what
we're
going
to
ship
to
users
that
was
really
important
to
have
that
tight.
Iteration,
though
one
of
the
other
most
important
things
that
we
did
at
this
stage,
which
see
it's,
you
know
it
doesn't
seem
like
infrastructure
work
at
all.
Is
we
branded
our
info
product
now?
This
was
kubernetes
right
kubernetes
at
github,
but
we
gave
our
platform
a
different
name.
A
We
said
this
is
Moda
and
we
made
it
clear
that
all
these
new
tools
and
patterns
were
a
part
of
Moda
at
github.
Moda
is
a
set
of
tools,
norms
and
documentation
and
kuben,
and
you
know
kubernetes
based
infrastructure
for
running
services
at
github
in
actuality,
all
of
those
elements
are
deployed
and
delivered
continuously
right.
We
work
on
em,
we
ship
them.
We
working
on
we
ship,
oh,
but
we
intentionally
developed
a
cadence
of
distinct
releases.
Each
release
of
Mota
comes
with
a
theme
set
of
features.
We
say.
A
Oh,
this
is
the
first
motor
release
it's
about
onboarding
it
it
comes
with
new
documentation,
it
comes
with
release,
notes
and
it
comes
with
quite
a
bit
of
fanfare
on
our
part
right.
We
make
a
big
deal
out
of
it.
This
establishes
our
products
narrative
right
and
specifically,
it
helps
us
to
highlight
resonance.
This
name.
Mota
gives
coherence
to
a
set
of
norms
and
tools.
It
helps
us
to
find
what
our
product
is
and
also
what
it
isn't.
This
first
release
of
muda
was
all
about
the
onboarding
experience.
A
Now
we
could
have
used
that
first
release
to
focus
on
where
we're
going
right.
We
could
have
put
a
stake
in
the
ground
for
what
we
think
a
service
platform
should
provide
or
require
when
it's
mature,
but
we
did
it
because
I
wouldn't
resonate
with
our
users.
Frankly,
they
don't
care,
they
just
want
to
get
their
apps
up
right.
We
met
them
where
they
were
at
at
that
point,
which
was
just
getting
started
and
we
listened
to
them.
A
We
use
product
ideas
to
focus
our
efforts
and
we
delivered
a
release
that
enabled
an
order
of
magnitude,
more
developers
to
begin
working
with
Moda.
So
what
comes
after
that?
Right
after
you
get
started?
What
comes
next
so
this
is
where
our
vision
comes
in,
and
this
is
an
exciting
step
for
us
getting
to
push
this
vision
out.
As
we
grow
our
platform,
we
learned
a
lot
about
the
failure
modes
of
underlying
kubernetes
infrastructure
and
different
ways
to
mitigate
those
failure
modes.
A
We
love
the
way
that
kubernetes
handles
servers,
server
failures
and
we
wanted
to
extend
that
idea
to
whole
clusters.
We
also
were
part
of
internal
conversations
about
extending
github
infrastructure
into
more
data
centers
more
regions.
What
would
that
entail?
What
would
a
bonerz
have
to
do?
What
does
failover
look
like
in
a
world
that
has
a
slightly
different
set
up?
Some
of
our
users
were
really
interested
in
those
conversations
as
well.
They
wanted
to
make
their
apps
as
reliable
as
possible.
A
A
It
also
helps
with
upgrades
right
to
have
a
have
a
flexible
concept
of
failure,
/
and
of
a
really
flexible
concept
of
resilience.
We
were
part
of
all
the
right
projects
to
start
laying
this
groundwork
that
could
be
used
when
the
other
parts
found
a
place
when
we
were
ready
to
roll
out.
We
invested
in
features
for
declaring
an
app
as
multi
cluster
capable,
so
how
do
users
think
about
routing
their
traffic,
about
deploying
to
multiple
places
about
getting
stats
for
multiple
deployments
right?
A
We
built
relationships
with
the
data
center
team
so
that
when
set,
you
know
that
we'll
be
setting
up
this
new
hardware
so
that
we
could
have
robust
communication
and
high
trust
on
both
ends.
We
prototyped
new
features
by
reaching
out
to
our
power
users,
who
are
already
interested
in
these
resiliency
concepts,
to
ensure
that
we
were
building
the
right
abstractions
and
got
their
insight
as
well,
so
the
payoff
to
that
I'm
going
to
admit
it.
It's
been
slow
right
in
some
form.
A
We've
been
investing
in
this
as
an
organization
for
over
a
year,
but
because
our
team
has
sort
of
called
our
shop
and
we've
named
our
vision,
we're
able
to
get
support
from
management.
We
not
only
have
a
vision
for
what
the
future
is
gonna
be
like,
but
we're
also
taking
responsibility
for
delivering
that
vision
and
for
delivering
that
future
and
for
ensuring
that
that
future
is
the
right
one
and
that
our
users
are
set
up
to
take
advantage
of
it
there.
A
There
are
short-term
wins
as
well
right
because
we're
treating
this
iteratively
each
release,
the
addressing
our
users.
Current
pain
points
also
moves
us
forward
towards
this
long-term
vision
of
resilience.
Our
last
release,
let
early
adopters,
opt
into
having
their
apps
work
in
multiple
clusters.
Our
next
one
is
going
to
include
tools
to
test
failure,
failure,
tolerance
on
the
app
level.
This
is
alongside
other
features
like,
for
example,
G
RPC
support,
which
we
found
out
was
important
by
getting
tickets
filed
by
various
teams
asking
for
it.
A
A
We
also
are
building
trust,
because
the
people
know
that
we're
working
the
problems
that
they
care
about
and
they're
excited
to
be
early
adopters
and
the
concept
from
product
development
is
that
early
adopters
are
really
key,
both
because
they're
active
participants
in
smoothing
out
the
product
experience
and
also
because
they
tend
to
have
a
high
degree
of
influence
over
future
users.
They
multiply
our
work
and
our
reach
in
ways
that
we
would
never
be
able
to
do
on
our
own.
So
how
are
we
gonna
wrap
this
up?
A
Well,
I
want
to
talk
about
some
of
the
challenges
that
we
face
in
this
approach
right.
How
do
we
support
all
these
devs?
You
know,
especially
as
we're
trying
to
balance
reactive,
incoming
problem
work
and
long-term
vision.
How
do
we
prioritize
the
work
that
we
want
to
do?
How
do
we
choose
what
features
we
want
to
build?
You
know
how
do
we
we
get
a
lot
of
feature,
requests
it's
not
as
simple
as
just
first-in,
first-out
or
anything
like
that.
Our
decisions
really
do
impact
other
people's
workflow
right.
So
how
do
we
do
that
respectfully?
A
And
how
do
we
do
that
with
the
maximum
amount
of
buy-in?
How
do
we
balance
the
needs
of
those
early
adopters
and
just
talking
about
those
power
users
with
newer
users
who
maybe
aren't
as
invested
in
this
platform
working
out,
and
how
do
we
save
today
compared
to
other
solutions
out
there
right,
because,
like
I
said
before,
you
know,
folks
will
sneak
away
and
you'll
be
like
what
is
that,
like,
oh
that's,
run
in
Heroku,
don't
worry
about
it.
How
do
we
make
sure
that
we
are
providing
the
things
that
that
the
that
folks
need?
A
So
these
questions
are
exactly
what
project
managers
answer
day
in
and
day
out
about
product
development
right,
if
you
develop
a
product,
all
of
this
should
be
really
familiar.
That's
really
exciting
right.
We
have
a
lot
to
learn
and
we
have
great
people
to
learn
it
from
actually
I'm
excited,
because
our
new
product
managers
sitting
in
the
audience
right
now
is
going
to
work
exclusively
on
that
with
us,
which
is
really
exciting.
A
We
have
a
lot
to
learn
from
people
who've
built
products
before
and
we
need
to
listen
not
just
to
our
customers,
but
we
need
to
listen
to
those
in
other
fields.
Right
infrastructures
of
field,
as
mentioned,
has
a
reputation
of
being
grumpy
and
homogeneous
and
hard
to
talk
to.
We
can
do
better
than
that
and
I
believe
in
us
and
I
know
that
I
know
that
we
can.
We
can
erase
that
bad
stereotype
and
that
bad
reputation
we
can
focus
on
humans
and
I
want
to
see
where,
where
that
focus
on
humans
could
take
us.
A
Finally,
I
would
love
to
hear
what
you're
doing
on
this
front
if
any
of
this
resonates
or
doesn't,
if
there's
things
that
you're
doing
at
your
organization
or
your
company,
please
come
and
talk
to
me
tweet
at
us
email
us.
Let
us
know
we
want
to.
We
want
to
spread
this
everywhere
and
thank
you
for
coming.