►
Description
Presented by John Arthorne, Shopify.
Our company was steadily growing the number of production services, to the point where we had several hundred running web applications, most of which had poorly defined ownership and ad-hoc infrastructure. These applications had a patchwork of tools such as build and deployment automation, monitoring, alerting, and load testing, but with very little consistency and lots of gaps. This talk will dive into our approach to infrastructure automation, which involved integrating our GitHub repositories with automation tools through a common infrastructure database. This automation tracks required actions for service owners through GitHub Issues, and generates pull requests for common configuration and software changes. This talk will summarize lessons learned and how to leverage GitHub APIs and web hooks to build similar infrastructure automation tools.
About GitHub Universe:
GitHub Universe is a two-day conference dedicated to the creativity and curiosity of the largest software community in the world. Sessions cover topics from team culture to open source software across industries and technologies.
For more information on GitHub Universe, check the website:
https://githubuniverse.com
A
So
just
to
give
you
a
rough
idea
of
the
flow
of
this
I'm
gonna
give
you
a
little
bit
of
background
on
the
company
and
our
tech
stack
really
quickly
just
to
set
the
the
the
baseline.
You
can
see
what
we're,
what
we're
talking
about,
describe
the
problem
that
we
were
facing
and
then
go
through
the
the
implementation
of
what
we
built
to
address
it
so
Shopify
for
those
who
aren't
familiar
is
a
build
software
as
a
service
for
commerce.
A
So
anyone
who
wants
to
have
an
online
store
or
a
physical
store
and
they
want
software.
For
that
we
provide
hosted
software.
We
have
about
just
over
half
a
million
merchants
on
the
platform
and
those
merchants
do
an
average
of
65
million
a
day
in
gmv
of
sales.
So
it's
you
know
fairly
small
compared
to
some
of
the
bigger
players
out
there,
but
you
know
non-trivial
size,
number
of
ploys.
A
So
you
know,
depending
on
where
you're
from
that's,
either
really
small
or
really
big
or
somewhere
in
between,
but
just
to
give
you
an
idea
of
where
we're
at
so
Shopify
started
out
actually
very
similar
to
github
around
the
same
time
as
github
as
a
rails
application
and
is
still
today
largely
based
on
a
one
very
large
scale,
rails
monolith,
which
is
the
core
back-end
platform.
However,
over
the
years
we've
ended
up,
building
more
and
more
pieces
surrounding
this.
A
This
monolith,
so
we've
over
time,
built
a
number
of
other
Ruby
and
rails
application
for
a
variety
of
different
purposes
from
you
know,
merchants
facing
and
critical
things
all
the
way
down
to
internal
tools
and
experiments.
So
lots
of
different
rails
applications
that
we
use
for
many
different
things.
We
also
have
a
fairly
large
data
stack
for
doing
sort
of
big
data
analysis
and
processing,
which
is
mostly
Python
based.
So
we
have
a
collection
of
applications
that
are
used
in
the
data
space
that
are
using
Python.
A
We
also
have
a
fair
number
of
systems
that
use
go-go
Lang's,
so
in
particular,
systems
where
concurrency
or
performance
are
super
important
concerns.
We
use
go
for
that
over
you
know,
put
Python
and
rails
or
breaks.
Ruby
are
very
expressive
languages,
but
they
not
always
the
fastest
languages.
So
we
have
some
of
that
in
the
mix
as
far
as
data
stores,
so
we
have
our
main
sort
of
relational
data
store,
back-end
uses
MySQL
and
with
Redis
for
like
ephemeral,
ephemeral,
key
value
stores.
A
We
have
a
number
of
other
different
storage
systems
as
well
all
or
most
of
this,
actually
everything
or
either
is
or
where
you
soon
will
be
containerized
applications
using
docker
and
orchestrated
with
kubernetes
and
running
either
inner.
We
have
our
own
physical
data
centers,
or
we
also
use
the
number
of
a
couple
of
different
third
third
party
data
centers,
where
we
have
a
number
of
these
applications
running.
A
So
the
main
problem
we
were
facing
as
I
was
saying,
is
that
you
know
we
had
these
sort
of
practices
within
the
company
that
were,
you
know,
evolved
around.
Maintaining
this
large
monolith
application-
and
we
were,
you,
know,
got
to
the
point-
we're
doing
quite
a
good
job
of
keeping
that
monolith
very
well
maintained
healthy.
But
then
all
of
this
other
stuff
that
had
grown
up
around
it
had
very
inconsistent
levels
of
infrastructure
supporting
it.
A
So
we
found
that
you
know
the
our
sort
of
way
of
working,
which
was
oriented
towards
a
very
small
number
of
application.
Just
wasn't
scaling
when
we
found
ourselves
in
a
company
where
we
had
you
know
a
couple
thousand
employees
and
hundreds
of
different
systems
floating
around.
So
we
knew
you
need
to
to.
You
know,
do
better
at
this,
and
in
particular
the
area
where
one
of
the
areas
where
we
were
suffering
was
in
what
I
call
service
infrastructure.
A
So,
if
you
think
about
you
know
the
analogy
of
a
production
system
like
a
race
car,
you,
you
might
think
superficially
that
it's
a
race
car
and
it's
driver
that
wins
races.
But
in
reality
it's
actually
a
huge
supporting
cast
around
that
car
there's.
You
know
designers
and
engineers
and
pit
crews
and
there's
there's
a
large
cast
of
supporting
characters
around
that
cart
and
that
enables
that
to
do
really
well
and
win
races
and
it's
very
similar
with
production
applications.
A
So
you
know
you
can
just
write
the
code
and
throw
it
over
the
wall
into
production
and
it'll.
They
might
do
okay,
it
may
or
may
not.
But
if
you
really
want
to
make
the
best
possible
production
system,
you
need
to
care
about
that
big
cast
of
supporting
characters
around
the
production
system
so
and
in
the
space
of
you,
know
we're
talking
about
a
production,
web
applications
or
or
or
systems
running
running
in
a
data
center.
A
So
there's
a
large
number
of
different
tools
that
you
need
in
order
to
a
really
good
job
at
maintaining
a
production
system
and
again,
all
that
is
well
and
good
if
you
only
had
like
one
or
two
production
systems,
but
if
you
find
yourself,
as
we
did
with
over
300
action
systems-
and
you
have
all
these
concerns
to
worry
about,
it
becomes
a
really
hard
problem
to
keep
track
of.
Like
do.
A
So
in
this
particular
case,
it
was
no
finding
the
list
of
apps
that
are
using
a
rails
version,
that's
about
to
become
obsolete
and
unsupported,
and
which
means
we
have
the
need
to
upgrade
them
or
shut
them
down.
So,
in
order
to
do
this,
we
had
to
gather
all
this
information
from
various
places.
Go
to
github,
find
all
the
source
code
find
where
the
runtimes
are
find,
who
to
talk
to
to
figure
out
what
we
need
to
do
with
this.
A
So
we
kind
of
had
to
manually
pull
together
all
this
information
to
make
these
decisions
or
make
these
changes
across
a
set
of
software
set
of
different
systems.
We
found
we
were
ended
up
doing
this
more
and
more,
and
this
was
clearly
not
an
approach
that
was
going
to
scale.
You
know
every
time
we
had
to
do
this.
It
took
days
out
of
developers
time
to
to
pull
this
information
and
make
all
these
decisions.
A
So
we
started
a
project
to
build
a
system
to
to
automate
this.
This
management
and
tracking
of
all
of
our
software
infrastructure
so
and
we
had
three
main
goals
in
building
this
project.
So
the
first
was
s
which
might
seem
weird
to
some
of
you,
depending
on
you.
Your
company
was
actually
establishing
ownership
for
production
systems,
so
we
we
had
a
culture
where
it
was.
You
know.
If
you
saw
something
wrong,
you
should
just
go
in
and
fix
it
and
it
we
didn't
actually
have
clearly
defined
like
people
or
teams
matched
to
any
different
system.
A
So
we
decided
this
was
something
that
we
needed
to
to
move
towards
a
more
concrete
model
of
of
who
owns
these
systems
and
what
that
means.
Second,
we
wanted
to
be
able
to
measure
by
how
well
are
we
doing
you
know
for
all
these
different
systems
and
are
we
getting
better
or
worse?
And
you
know
what
do
we?
You
know
how
much
work
do
we
have
to
do
to
get
better
at
this?
And
third,
we,
you
know
all
of
these
manual
repetitive
tasks
for
setting
up
setting
up
monitoring
or
moving
your
logs
to
Splunk.
A
You
know
for
one
or
two
services,
it's
not
a
big
problem,
but
when
you
have
dozens
and
dozens
of
services,
all
of
these
kind
of
infrastructure
concerns
become
like
it
just
doesn't
scale
right.
You
have
to
do
all
this
manual
toil
work
to
integrate
all
these
different
things
all
the
time
and
we
felt
by
having
all
the
information
pulled
together.
We
might
be
able
to
automate
some
of
these
things,
so
we
built
this
okay,
so
gonna
go
through
these
different
goals,
so
the
first
one
as
I
mentioned
so
Shopify.
A
You
know,
as
a
small
start-up
had
this
culture
of
like
one
of
the
company
mottos
is
act
like
an
owner,
which
means,
if
you
know,
if
you
see
something,
needs
to
be
doing.
You
know
pretend
you
know.
If
I
was
the
owner
of
this
company,
what
would
I
do?
Okay,
do
that,
like
that
was
the
solution
to
most
things
was,
do
what
you
would
do
if
this
was
your
company
and
you
were
responsible
for
success,
and
that
was
an
approach
that
really
worked
well
in
a
very
small
company.
A
But
when
you
have
600
people
being
told
to
act
like
an
owner,
you
got
that
bystander
effect
right
where
everyone
you
know,
feels
that
they're
1/8
they're
able
to
reach
in
and
do
it,
but
who
is
the
one
who's
actually
gonna,
go
and
make
some
change
or
do
something
for
any
system.
So
there's
number
of
different
cases
where
not
having
these
clearly
defined
owners
was
clearly
becoming
painful
for
us.
So
one
example
is
when
we
have
an
incident
so
something's
down,
and
we
urgently
need
to
find
out
more
information
about
you
know.
A
A
You
know
it's,
it's
all
well
and
good
that
I
can
reach
in
and
open
to
open
a
PR
against
some
other
app
and
it
gets
accepted,
and
it's
all
good,
but
if
I
want
to
make
these
long-term
decisions,
I
really
wanna
have
a
conversation
with
someone
who
knows
the
background
of
that
service
and
can
say
like
here's.
This
thing
I
want
to
do
with
it.
You
know
what
does
that
make
sense
for
the
goals
of
this
application?
A
Should
we
build
some
new
thing
that
replaces
both
of
them
or
whatever
you
know
you,
you
really
need
somebody
to
talk
to
you
to
have
those
decisions
about
making
long-term
making
long-term
decisions
about
the
different
systems
you're
running,
so
we
decided
what
we
really
need
was
defining
ownership
for
things
and
moving
more
towards
a
concrete
notion
of
like
this
system
is
owned
by
this
team.
However,
you
know
we
really
wanted
to
avoid
the
other
end
of
the
spectrum
of
this.
A
You
know,
if
you
see
in
very
large
companies
and
in
governments
where
you
have
a
very
authoritarian
notion
of
like
who
owns
what,
where
you
can't
even
access
the
source
code
for
other
things.
You
know
and
it's
very
bureaucratic
and
very
slow,
and
we
knew
very
much.
We
didn't
want
that.
We
wanted
to
have
the
you
know,
as
as
github
users
you're
all
familiar
with
the
power
of
being
able
to
reach
in
and
open
a
pull
request
against
things
and
and
have
the
flexibility
to
do
that
when
you
need
to.
A
We
wanted
to
make
sure
we
kept
that,
but
you
know
giving
people
the
level
of
autonomy,
but
still
having
a
clearly
defined
notion
of
ownership
for
things
so
this
over
a
year
ago.
So
we
had,
we
decided.
We
need
to
do
something
about
this
and
we
we
said,
what's
the
simplest
possible
thing,
we
could
do
as
a
prototype
of
this
to
flush
out
the
ideas
in
the
tools.
A
So
if
we
were
starting
this
today,
we
would
probably
use
the
code
owners
file
that
has
been
introduced
since
we
started
on
this.
Our
version
of
moverá
simplified
version
of
the
code
owner's
file
that
is
available
in
github
today,
but
just
very
simple.
It's
just
this
team
owns
this
repo,
not
nothing
fancy,
and
it
turned
out
that
you
know
we
actually
only
intended
this
as
a
prototype
like
minimal,
viable
solution
to
this,
but
it
actually
worked
really
well.
A
So
one
of
the
cool
advantages
is:
if
you
have
a
system
where
you
need
to
change
owner
someone,
the
team's
get
rear
door,
someone
leads
the
company
or
someone
joins.
You
can
just
open
a
pull
request
and
say:
hey
I
think
we
should
move
ownership
of
this
to
that
team.
What
do
you
think
you
know
tag
the
previous
owner
for
review
on
that
change,
get
it
approved
and
ship
it?
A
It's
kind
of
neat,
so
the
second
goal
we
had
was
measuring
like
how
well
are
we
doing?
Are
we
getting
better
at
worse
at
like
that,
all
this
infrastructure
that
we
need
around
these
applications?
If
you're
a
team
that
has
a
large
number
of
applications?
You
probably
want
to
know
like
where
should
I
vote
spend
my
time
which
of
my
services
are
doing
really
well
and
which
ones
need
work?
So
I
can
figure
out
where
to
spend
my
time
and
we
also
decided
we
needed
to
you
know
we
have
this
core
monolith.
That's
super
important.
A
Has
you
know
millions
of
dollars
G&B
going
through
it,
and
then
we
have
these.
You
know
little
toy
internal
tools
where,
if
it
goes
down,
it's
not
a
big
deal,
a
one-size-fits-all
solution.
You
know,
I,
have
these
two
rails
app
was
vastly
different
requirements.
It
doesn't
make
sense
to
have
the
same
infrastructure
around
these
two
different
apps,
so
we
knew
we
needed
to
sort
of
differentiate
between
these
different
systems
and
some
in
some
way.
A
A
So
the
very
first
benefit
of
this
was
we
could
create
this
dashboard
for
every
production
system.
There's
this
page.
Now,
where
you
can,
you
can
search
for
your
system
and
pull
up
this
page,
and
you
can
see
you
know
description
of
what
it
is
who
owns
it?
What
is
the
public
URL
where's
the
source
code,
where
the
Splunk
logs
where's,
the
you
know,
pager
duty
team?
A
That's
on
call
for
this
thing,
so
you
sort
of
just
have
all
this
information
pulled
together
in
one
place
was
a
huge
benefit
for
us,
where
we
no
longer
had
to
jump
around
between
all
these
different
places
to
find
out
information.
We
can
just
go
to
this
one
place
and
access
all
this
information
we
needed.
You
can
see.
We
also
pulled
in
a
number
of
different
metrics,
so
from
github
we've
taken
commits
stats
and
issue
issue
data
and
captured
that
as
well,
so
we
can
kind
of
see
over
time.
A
A
So,
okay,
so
we've
got
all
this
information
like.
Is
that
good,
like
is
this
system,
got
everything
that
it
needs,
so
we
know
what
it
has
does
it
need
to
have
more
or
it
is
in
the
order?
Does
it
have
too
much
in
place,
so
we
decided
we
need
to
create,
create
differentiate
between
the
systems
and
create
these
different
tiers
of
applications
where,
as
you
go
up
in
tears,
there's
higher
and
higher
expectations
of
what
you
need
for
this
for
this
system.
We
wanted
to
this
in
a
fairly
lightweight
way.
A
We
didn't
want
to
have
a
lot
of
process
around
this,
so
we
tried
to
keep
it
as
simple
as
possible.
So
what
we
came
up
with
this
is
this
four
tier
system
we're
at
the
very
bottom
we
have
tier
4
is
experiments,
and
you
know
we
have
a
culture
in
the
company
of
highly
encouraging
people
to
do
experiments.
If
you
want
to
spin
up
a
new
system
to
try
something
out,
go
ahead,
you
know
no
permission
needed
just
do
whatever
you
want
and
we
want
to
keep
the
bar
very,
very
low
for
that.
A
We
don't
want
to
impose
a
lot
of
requirements
on
a
system
like
that,
because
the
whole
point
is
moving
fast
and
trying
things
out
so
for
experimental
systems.
We
only
have
two
requirements,
one
that
there
be
an
owner
and
that
it
resolves
security
issues
so
very,
very
low
bar
and
then,
as
we
go
up
so
the
next
tier
of
applications
are,
for
the
most
part,
internal
tool,
so
the
things
that
are
that
are
useful
within
the
company,
but
not
not
super
critical
for
these
ones.
A
What
we
want
to
make
sure
is,
we
haven't,
have
all
the
infrastructure
in
place
so
that
a
team
of
people
can
work
on
it.
So,
most
importantly,
it
has
you
know
automation
for
setting
up
my
local
Deb
environment.
It
has
a
CI
in
place.
It
has
very
basic
monitoring
in
place.
So
it's
something
that
anyone
in
the
company
can
come
in.
They
can
do
a.
They
can
do
a
clone
of
this.
A
So
tier
two,
our
systems
that
are
you
know
very
important
where,
if
it's
down
either
there's
some
minor
impact
to
our
customers
or
there's
a
major
impact
internally.
As
you
know,
a
large
number
of
people
in
the
company
can't
get
the
work
done
because
they
need
this
system
to
be
to
be
up.
So
for
the
for
this
level,
we
have
another
number
of
other
requirements
that
we
add
on
the
most
important
ones.
A
Are
that
there
be
someone
on
call
for
it,
so
the
developer
team
that
is
building
this
has
an
on-call
rotation
for
it
and
that
there's
monitoring
a
place.
Would
that
will
alert
when
it's
down
and
it'll
notify
that
team
and
a
number
of
other
like
fairly
minor
requirements
compared
to
that
and
then
a
bit
very
top?
We
have
a
very
small
number
of
systems
like
our
core
monolith
that
are
Tier
one.
A
These
are
things
that
are
super
critical
to
the
company
if
it
goes
down,
there's
either
a
large
financial
or
reputational
impact,
and
we
absolutely
need
these
to
be
our
highest
quality,
most
resilient
systems,
and
so
we
have
a
number
of
other
requirements
on
on
the
top
tier
systems.
The
biggest
one
is
that
it
have
be
resilient
to
any
single
data
center
going
down,
so
any
system
at
Tier
one
should
be
able
to
withstand
us
any
one
data
center
going
down,
and
we
should
be
able
to
do
withstand
that
through
through
resiliency
and
redundancy
okay.
A
So
now
that
we
have,
we
know
the
list
of
all
of
our
things.
We
know
what
tier
it's
in
now
we
can
create
this
scorecard
view
can
show
you
you
know
for
an
individual
or
a
team.
You
can
get
this
list
of
here's
all
my
things.
Here's
all
the
requirements
for
the
system
of
that
kind,
and
how
are
you
doing
so?
You
can
quickly
go
to
this
page
and
go
okay.
You
know
some
of
these
things.
Look
pretty
good,
but
this
thing
at
the
top
has
got
a
bunch
of
things.
A
Miss
I'm
gonna
go
in
and
fix
that,
so
it
gives
teams
the
information
they
need
to
make
decisions
about
what
they
need
to
be
working
on
for
the
infrastructure
of
their
applications.
So
this
is
a
really
useful
tool.
The
second
thing
we
added
we
actually
added
just
over
this
summer
as
an
experiment,
so
we
thought
you
know
we
just
had
a
little
bit
of
gamification
into
this.
You
know
this
is
really
this
works
really
well
and
in
video
games.
Let's
try
this
out,
so
we
created
a
leaderboard
view
and
literally
all
we
did
was
recruited.
A
This
view
it
ranked
every
team
in
the
company
all
how
many
systems
and
how
many
you
know
things
are
missing
and
Cave
and
I
gave
him
this
percentage
score,
and
we
just
sent
an
email
saying
like
here:
we've
got
week
rid
this
view
like
hats
off
to
these
group
of
people
at
the
top.
That
did
a
great
job.
You
know
here,
you
can
go
and
see
how
you're
doing
and
that's
all
what
we
did.
We
didn't
do
any
enforcement
or
any
like
anything
beyond
that.
A
Just
doing
that,
one
thing
was
the
had
the
single
biggest
impact
on
the
quality
of
all
of
our
systems.
So,
within
a
month
of
just
creating
this
view
and
letting
people
know
it
was
there,
tons
of
developers
were
just
like
you
know
if
they
saw
a
yellow
or
red,
they
were
just
like.
You
know
it
just
made
them
anxious
and
they
were
like
I
gotta.
You
know,
I've
got
a
competing,
it's
the
other
team
over
there
and
I
want
to
do
really
well.
A
This
was
like
an
intrinsic
motivator
for
a
lot
of
developers
to
up
their
game
on
their
systems
and
do
a
lot
better
job
without
any
further
prodding
just
by
having
this
this
view
available.
So
that
was
really
interesting
discovery
for
us,
so
so,
once
we've
got,
you
know
we're
measuring
how
well
we're
doing
the
next
step.
As
I
mentioned
the
beginning
was
now.
We've
got
all
this
information
and
we've
got
all
this
like
connections
in
place
between
our
different
systems.
A
Can
we
build
some
automation
to
make
it
as
easy
as
possible
to
maintain
systems
and
take
away
some
of
the
manual
toil
involved?
So
this
is
where
it
gets
really
really
fun
for
developers
where
they
can
see
this,
not
just
as
a
tool.
That's
telling
them
what
they're
doing
wrong,
but
it's
it's
a
tool,
that's
helping
them
figure
out
how
to
fix
how
to
fake
and
how
to
improve
their
systems.
So
the
first
thing
we
can
do
with
service
DB
is:
we
can
open
github
issues
for
and
anything
that
is
wrong
with
that
system.
A
So
will
automatically
open
an
issue
that
says
here's
the
thing:
you're
missing:
here's
a
link
to
some
documentation
to
help
you
figure
out
what
to
do
about
it,
so
it
gives
developers
all
the
information
they
need
to
to
to
make
a
change
to
their
system
and
even
more
valuable
II.
You
know
robots
opening
issues
against.
You
doesn't
sound
all
that
cool,
but
robots,
that'll
automatically
close
issues
when
the
underlying
problem
is
fixed,
really
is
really
valuable
right.
A
So
this
this
bot
will
now
monitor
this
system
to
make
sure
this
case
it
was
it's
missing,
uptime
monitoring,
so
once
we
detect
that
uptime
monitoring
is
in
place
for
this
system
will
automatically
go
and
close
the
issue
for
you,
so
we
can.
We
can
make
sure
that
you
know
we
do
that
bookkeeping
and
make
sure
that
you
know
we
don't
leave
a
mess
laying
around
in
your
issue
tracker
for
things
that
are
no
longer
a
problem.
A
So
the
next
thing
we're
able
to
do
by
having
all
this
information
together
was
what
we
call
like
fighting
against
the
email
bots.
So
we
have
a
lot
of
different
third-party
services.
We
use
many
of
them
send
alerts
to
us
in
the
form
of
emails
to
say
either
you've
exceeded
our
quota
or
this
thing
is
wrong
or
we're
doing
a
maintenance
FYI
and
once
we're
at
the
scale
of
hundreds
of
systems.
These
were
just.
They
just
became
noise
like
nobody
was
reading
these
emails
anymore
because
there's
just
too
many
of
them.
A
So
what
we
did
was
we
we,
instead
of
having
these
emails,
go
to
a
person
we
sent
all
these
emails
to
a
service
of
DB
and
what
it
does
is
it
parses
all
the
emails
that
come
in
it
assesses
like
what
needs
to
be
done
about
this.
Is
this
just
like
an
FYI
think
I
can
just
throw
it
away,
or
is
this
something
that
requires
action
by
team
and
if
it
does,
what
we
do
is
we
take
the
body
of
that?
Essentially,
we
turn
it
into
a
github
issue.
We
know
what
repo
it's
against.
A
We
can
open
an
issue.
We
can
tag
that
team
because
we
know
what
team
owns
it
and
that
way,
we've
sent
this
information
that
was
sort
of
blasted
out
to
all
of
our
admins.
We've
now
filtered
all
this
information
to
the
team
that
actually
needs
to
know
that
problem,
so
they
can
fix
it.
So
this
was
really
a
huge
benefit
and
the
noise
of
these
sort
of
email
based
alerts.
We
were
getting
so
the
next
thing.
Next
thing
we
built
using
this
infrastructure
was
automating
some
of
these
repetitive
tasks.
A
So
you
know
a
simple
one
here
is
you
know?
This
is
a
system
that
that
should
have
uptime
monitoring,
but
it
doesn't
but
hey.
We
know
what
the
public
ural
is
we
and
we
have
access
to
the
we.
So
we
use
a
third-party
tool
called
Pingdom
for
doing
uptime
monitoring,
which
has
api's.
We
could
set
that
up
for
you.
So
just
say:
yes,
I
want
this
click
click
the
fix
button
and
we
automatically
set
up
the
monitoring
for
that
system
for
you.
So
it's
a
number
of
other
places
where
we
we
built.
A
My
automation
like
this,
that
would
that
just
took
away
a
lot
of
manual
pain.
Another
big
one
was
wiring
all
of
our
alerts
into
pager
duties
so
that
you
know
when
this
alert
fires,
the
appropriate
team
and
paid
judy
was
was
alerted.
Setting
those
things
up
is
a
real
pain,
I'm
sure,
if
any
of
you've
done
it
you,
you
know
what
I'm
talking
about
that
you
have
to.
You,
have
to
sort
of
go
between
these
different
tools
and
copy
an
authentication
key
here,
and
there
there's
many
different
ways
to
do
it:
lots
of
different
ways.
A
You
can
do
it
wrong,
it's
just
something
that
you
don't
want
to
have
to
bother
to
do
so.
This
was
another
thing
where
we
were
able
to
use
automation
to
say:
look,
we
know
what
you're
monitoring
is.
We
know
what
the
pay
duty
team
is
for
this
particular
system.
We
can
configure
all
those
alert
connections
for
you,
and
that
was
a
really
valuable
benefit
of
having
all
this
information
together
so
where
it
gets
really
fun,
as
we
started
to
do
automatic
code
authoring
for
simple
changes
for
these
systems.
A
So
is
this
started
out
with
a
lot
of
our
systems
are
using
kubernetes,
which
have
these
configuration
files
that
have
you
know
various
information
about
the
infrastructure
for
this
system
we're
able
to
open
a
pull
request
whenever,
let's
say
we
need
you
to
to
change
to
a
different
like
we
need
some
configuration
to
change
to
be
made
across
a
whole
fleet
of
systems.
We
can
automatically
open
pull
requests
to
do
that
for
you,
and
that
was
a
really
valuable.
A
Just
to
save
people
from
having
to
manually,
you
know,
apply,
updates
to
things
and
so
many
opportunities
to
get
things
wrong.
The
other
area
that
we've
been
working
on
over
the
summer
is
doing
automatic
software
updates
for
your
dependencies.
So
this
sounds
a
little
bit
like
what
was
announced
yesterday,
actually
with
the
dependency
graph
and
the
security
alerts
where
we
can.
A
You
know
we
know
that
using
this
library,
if
you
want
to
update
this
library
or
if
there's
a
strong
reason
to
update
this
library,
we
can
open
a
pull
request
for
you
automatically
to
make
that
change
and
then
see.
I
will
run
and
you
can
look
at
that
and
you
might
be
able
to
just
merge
it
and
you'll
be
ok.
So
this
just
took
away
a
lot
of
manual
toil
that
developer
teams
across
the
company
we're
having
to
do
to
maintain
their
systems.
A
So
you
can
kind
of
see
we
turn
this
on
in
June
and
like
this
is
the
contribution
graph
for
the
for
the
service.
Db
github
account
and
you
can
see-
became
much
more
active.
It's
now,
one
of
our
more
productive
developers
in
the
company,
because
it's
doing
a
lot
more
of
these
repetitive
tasks
for
us,
so
I
can
go
a
little
bit
into
more
detail
on
the
implementation
quickly.
So
the
system
is,
as
I
mentioned,
beginning
it's
a
rails
application
and
we
connect
out
to
the
api's
of
these
different
systems.
A
So
we
use
the
github
API
to
access,
so
some
of
the
really
core
models
of
the
repos
users
and
teams
and
the
issues
that
are
open
against
these.
These
repos.
We
also
connect
to
the
different
runtimes,
pull
that
information
in
and
connect
to
all
the
different
third-party
tools
and
pull
that
information
into
our
database.
We
run
into
pretty
severe
rate
limiting
problems
with
with
github
and
most
of
these
other
systems.
A
So
the
general
approach
we've
taken
here
is
that
service
DB
will
sort
of
copy
all
this
information
in
periodically
into
our
database,
and
then
we
don't
actually
call
out
to
github
or
to
these
third-party
systems
during
routine
usage
of
the
tool
or
when
we're
running
our
checks
against
the
our
systems.
We
sort
of
rely
on
our
internal
database.
We
just
reach
out
when
we
need
to
to
access
new
information.
A
Github,
of
course,
has
web
hooks,
which
are
extremely
valuable
so
that
we
can
really
cut
down
on
the
number
of
times.
We
have
to
go
and
ask
for
information.
They'll
tell
us
when
things
have
changed
that
we
care
about,
and
we
can
update
our
model
sort
of
incrementally,
so
that
was
really
valuable
for
us.
Some
of
these
systems
webhooks
some
of
them
some
of
them
don't
so
as
far
as
the
user
interface
for
the
developer
team,
so
we
have
a
web
application.
You've
seen
some
of
the
screenshots
of
the
different
parts
of
that.
A
We
also
have
a
slack
bot
that
connects
into
this
as
well.
That
gives
you
a
number
of
different
tools
that
are
available
for
women
from
within
slack,
which
is
really
useful.
So
a
couple
of
the
other
key
concepts
here
in
the
system.
So
we
have
these
classes
called
checks
which
are
each
check.
Is
it's
a
piece
of
logic
that
you
know
checks
whether
this
service
satisfies
this
requirement
like
it
has
an
owner?
It
has
uptime
monitoring,
etc.
A
One
of
the
neat
ones.
That
was
really
valuable
was
this.
We
have
this
ssl
check
and
what
it
checks
is.
Is
this
system
using
https
protocol
and
is
the
certificate
for
that
valid
and
not
near
expiry?
So
that's
a
really
valuable
check
that
were
able
to
run
against
all
the
systems
and
let
you
know-
and
then
we
have
these
other
things
that
are
also
sources
of
information
that
we
were
pulling
in,
like
the
emails
from
the
third-party
providers,
and
we
also
have
slack
commands
that
people
can
run
to
say.
A
I
want
to
sort
of
generate
an
event
to
say:
oh
everyone
using
every
tier
for
app
in
this
kubernetes
cluster
needs
to
do.
X-
and
we
can
generate
events
for
that,
so
events
in
the
middle
are
so
we
have
all
these
things
that
are
sources
of
sort
of
sort
of
alerting
on
our
infrastructure
that
letting
us
know
that
something
is
wrong.
The
event
has
a
logic
to
decide
what
to
do
about
that.
So
this
is
something
we
have.
We
spent
a
lot
of
time,
refining
because
it
can
be.
A
It
can
very
quickly
get
very
noisy
if
you
have
systems
that
are
automatically
opening
issues
against
you
automatically
pinging
you.
We
really
want
to
make
sure
that
we're
only
sending
an
alert
to
a
team
when
it's
important
and
that
we're
using
a
channel
that's
appropriate
depending
on
the
severity.
So
if
this
is
something
that's
high
urgency
we'll
use
the
stock
announcement
cuz,
that's
the
most
reliable
way
to
access
our
dev
teams
quickly.
If
it's
something
that's
important
but
not
urgent,
like
it's
something
I
need
you
to
do
soon,
but
not
right.
Now.
A
Github
issues
are
the
best
way
to
do
that,
where
they
can
take
action
on
that.
You
know
whenever
they
whenever
they
can
fit
it
into
their
development
cycle,
and
of
course
you
know
ideal
case
is
we
can
omit
a
pull
request
for
those
for
those
situations,
and
you
know
and
give
you
a
first
cut
at
what
the
fix
might
look
like.
A
So
we
have
these
various
channels
and
we'll
pick
the
one
that's
most
appropriate,
depending
on
the
the
check
or
the
different
alert
that's
coming
in,
and
the
events
are
the
the
bits
of
logic
that
make
those
decisions
so
getting
a
little
bit
into
the
library
upgrades,
which
is,
you
know,
similar
to
what
was
announced
yesterday.
So
this
is
another
thing
that
can
very
quickly
get
it
get
noisy
right.
So
if
you
have
a
lot
of
systems
and
the
each
of
them
has
dozens
or
hundreds
of
dependencies,
we
don't
want
to
be
opening.
A
Even
if
it's
a
pull
request,
we
don't
want
to
be
opening
them
every
time.
Every
single
third-party
dependency
use
has
a
new
version
available,
because
it's
just
too
noisy
like
on
most
systems.
That
would
mean
all
you're
doing
all
day
is
upgrading
all
these
little
libraries.
So
what
we
wanted
to
do
is
make
sure
that
we're
only
sending
opening
these
pull
requests.
One
that's
really
valuable
to
do
so.
So
we
have
three
different
sources
for
you
know
that
that
tell
us
when
it's
important
to
open
these
pull
requests.
A
So
the
first
one
most
obvious
one
as
mentioned
yesterday,
when
does
the
security
vulnerability
advisory
against
a
particular
library,
then
in
those
cases,
is
very
urgent
that
we
make
sure
that
every
system,
that's
using,
that
version
of
that
library
gets
upgraded
as
soon
as
possible.
So
that's
the
first
one
that
we
implemented
that
we
open
pull
requests
immediately
as
soon
as
an
advisory
comes
out
with
the
upgrade
of
that
of
the
library
in
question.
A
Another
one
that
we
built
was
we
have
this
notion
of
being
able
to
deprecate
a
particular
library
or
library
version
across
the
company,
and
this
is
something
that
we
use
for.
We
have
a
number
of
internal
libraries
that
are
that
are
used
by
many
different
systems
and
we
might
decide
we
don't
want
to
maintain
this
anymore
or
there's
a
compelling
reason
why
we
need
you
to
upgrade
it
or
it
could
be
a
third-party
library
where
we
have
a
important
company
reason
to
say
you
should
not
ever
use
this
version
of
this
library
anymore.
A
Please
upgrade
it
so
we
have
essentially
a
company
blacklist
of
library
versions
where,
if
you're
using
that,
will
automatically
open
up
or
request
to
get
you
to
fix
it.
The
third
category
is,
we
have
this
notion
of
important
libraries,
so
this
is
for
rails
applications.
We've
identified
about
a
dozen
libraries
that
are
just
really
critical
to
keep
updated
like
these
are
libraries
where,
if
you
don't
update
it,
chances
are
you're
going
to
be
very
quickly
running
to
you
know
a
security,
vulnerability
or
you're
going
to
be.
A
You
know
just
creating
a
lot
of
tech
debt,
because
this
is
this
is
a
library
that
you're
you
we
are
going
to
have
to
upgrade
sooner
rather
than
later,
so
we
we
might
as
well
open
a
PR
for
it.
So
this
is
where
we're
trying
to
find
the
balance
of
not
blasting
you
with
PRS
for
every
little
tiny
third-party
library
you're,
using
just
making
sure
we
open
these
when
it's
really
when
it's
really
valuable.
So
we
have
had
this
notion
of
these
libraries
that
you
should
always
keep
up
to
date.
That's
that's
fairly
small!
A
So
all
of
these,
these
are
the
three
different
sources
for
the
auto
generation
of
pull
requests
for
updating
those
libraries.
So
some
of
the
future
directions
were
looking
at.
We've
got
lots
of
different
ideas.
We
definitely
like
to
automate
more
infrastructure
work,
so
any
kind
of
automatic
work
that
people
need
to
do
to
run
their
systems,
we're
always
on
the
lookout
for
those
so
that
we
can
add
more
sort
of
one-click
automation
to
to
set
up
or
update,
or
whatever
your
infrastructure
we'd
like
to
do
library
upgrades
for
other
languages.
A
So
today
we
only
support
rails
and
we've
got
some
support
in
place
for
for
node,
but
we'd
like
to
be
able
to
do
that
for
golang
and
python
as
well.
So
that's
something
that
we
we're
hoping
to
do,
or
maybe
we'll
be
able
to
take
advantage
of
some
of
the
capabilities
that
github
is
introducing
for
that
through
their
dependency
graph.
Another
area
that
we're
want
to
work
on
is
defining
SL
O's
service
level
objectives.
A
We
have
a
lot
of
whenever
we
have
any
kind
of
service
disruption.
We
have
a
post-mortem
and
we
come
up
with
a
set
of
actions
that
need
to
be
done.
These
are
invariably
opened
as
github
issues
sort
of
all
over
the
place
in
the
company,
depending
on
where
the
problem
is,
and
we
have
no
sort
of
single
way
to
keep
track
of
all
these
action
items.
And
how
are
we
doing
at
making
sure
that
we
do
the
follow-up
whenever
we
have
a
failure?
A
A
So
the
key
takeaways
from
us
out
of
all
this
that
you
know
I
think
you
should
think
about
like
is
this
applicable
to
me
or
how
can
I?
How
can
I
do
something
similar
in
my
company,
so
the
notion
of
ownership
was
something
that
was
a
big
cultural
change
for
us,
like
moving
away
from
the
startup
mentality,
if
everyone
just
reaches
in
and
does
what
they
need
to
do,
to
get
things
done
to
a
model
where
you
can
do
that,
but
there's
also
someone
to
find
who's,
this
sort
of
steward
or
owner
for
the
system.
A
A
So
when
you
have
one
or
two
applications
manual
toils,
not
so
bad,
but
when
you
have
dozens
or
hundreds
these,
these,
this
toil
work
and
spread
across
a
large
engineering
organization,
really
adds
up,
so
we've
become
hypersensitive.
Now
to
hey
like
this
is
like.
Why
are
we
opening
an
issue
telling
people
to
manually
do
X
like?
Can
we
build
some
automation
to
do
that
for
them
and
that's
something
that
we
we
find
ourselves
paying
a
lot
more
attention
to
now?
A
So
that's
about
it
for
my
talk,
so
I'm
not
gonna,
be
able
to
take
any
more
questions
right
now,
but
I
feel
free
to
catch
up
to
me
later
today
or
I'll
be
at
the
back
of
the
room.
If
you
have
more
questions
or
you
can
act
me
on
on
social
media,
so
thank
you
much
for
your
attention
and
enjoy
the
rest
of
the
conference.