►
Description
Google has been practicing Site Reliability Engineering (SRE) since the early 2000’s but only recently began speaking about it publicly. Some have called SRE “DevOps 2.0,” while others view it as a competing standard. Is SRE the new DevOps? Or is SRE something only large companies can benefit from?
Attendees will learn how SRE and DevOps align, diverge, and disagree.
DevOps vs. SRE → http://bit.ly/2TRsJXK
Watch more:
Next '19 DevOps & SRE Sessions here → https://bit.ly/Next19DevOpsSRE
Next ‘19 All Sessions playlist → https://bit.ly/Next19AllSessions
Subscribe to the GCP Channel → https://bit.ly/GCloudPlatform
Speaker(s): Seth Vargo
Session ID: OPS100
product:DevOps & SRE; fullname:Seth Vargo; event: Google Cloud Next 2019; re_ty: Publish; product: Cloud - General; fullname: Seth Vargo;
A
A
So
with
that,
let's
go
ahead
and
get
started
in
the
beginning,
there
were
two
groups
of
people
on
the
Left
we
have
developers.
Developers
are
concerned
with
agility.
They
want
to
build
features,
build
software
and
ship
it
as
quickly
as
possible
to
get
it
in
the
hands
of
customers
and
users
on
the
right
hand,
side.
We
have
operators,
operators
are
concerned
with
stability,
it's
not
broken,
please
don't
touch
it,
and
this
wasn't
just
a
personality
thing.
This
was
driven
by
the
business
as
an
operator.
A
A
Every
new
line
of
code
we
write
has
the
ability,
as
the
potential
to
have
a
bug,
has
the
potential
to
have
a
performance
regression.
So
these
are
directly
competing
ideas.
We
have
developers
who
are
trying
to
move
quickly
and
introduce
instability
and
operators
who
are
trying
to
slow
things
down
as
much
as
possible
because
it's
not
broken.
Please
don't
touch
it,
but
here's
where
things
get
a
little
bit
interesting
developers
work
closer
to
the
business,
both
physically
and
metaphorically
developers
often
sat
in
the
same
building
as
the
directors
and
the
vice
presidents
and
the
CEOs.
A
They
were
physically
located
closer
to
the
business
decision
makers.
Operators,
on
the
other
hand,
were
often
in
a
data
center.
Their
desk
might
be
hundreds
or
thousands
even
miles
away
from
the
corporate
office.
They
often
felt
disconnected
they
felt
like
their
ideas,
weren't
being
heard
educationally.
These
two
groups
of
people
often
came
from
different
backgrounds
developers
traditionally
had
a
software
engineering
background,
computer
science,
information
systems,
computer
engineering,
something
along
those
lines
from
a
two
or
four-year
degree
or
more
operators
tended
to
come
more
from
a
practical
background.
A
They
might
have
like
an
associate's
degree
or
a
practicum
and
something
like
network
engineering,
so
there's
a
skills
gap
on
both
sides.
Developers
are
really
good
at
algorithms
and
writing.
Software
operators
are
really
good
at
understanding,
network,
topologies
and
failure
scenarios
and
how
redundant
does
a
SATA
Drive
actually
need
to
be
in
order
to
have
this
many
nines
of
availability,
but
because
developers
were
closer
to
the
business
oftentimes,
they
would
just
write
their
code
and
they
would
throw
it
over
the
wall
to
the
operators.
A
That
animation
was
nice
right.
Yeah
I
worked
really
hard
on
that,
so
these
developers
would
throw
their
code
over
to
the
operators.
They'd
be
like
here's,
my
IP
HP.
Please
go
run
it
for
me.
Things
have
a
great
day
and
these
operators
remember
they.
They
don't
have
a
traditional
background
and
software
engineering.
They
may
have
never
worked
with
these
languages
before,
in
addition
to
their
responsibilities,
of
keeping
the
network
up
and
running,
making
sure
hard
drives
are
not
failing,
making
sure
that
servers
are
not
faulting.
A
They
now
have
to
understand
bugs
in
the
application,
because
when
that
application
goes
down,
they're
not
paging
the
developer.
They're,
getting
paged
they're
getting
woken
up
in
the
middle
of
the
night
for
a
bug
in
the
application
code,
but
because
developers
were
closer
to
the
business
when
operators
would
complain,
no
one
would
listen.
A
So
the
DevOps
movement,
really
in
its
purest
form,
is
about
breaking
down
that
wall
between
developers
and
operators.
I
know
that
was
another
nice
animation
by
breaking
down
the
barriers
between
developers
and
operators
and
aligning
the
incentives
between
them.
We
can
deliver
software
better
and
faster
and
more
safely.
For
our
end
users.
A
What
are
some
ways
that
we
can
do
this,
though?
Well
one
of
the
easiest
ways
that
you
can
break
down
the
barriers
between
developers
and
operators
is
to
put
them
in
the
same
physical
room.
This
has
been
shown
to
work
successfully
time
and
time
again.
It's
a
time-tested
pattern.
Instead
of
having
your
operators
go
sit
in
the
data
center,
hundreds
of
miles
away
put
them
in
the
same
room
as
the
developers
make
them
attend.
A
Stand-Ups
make
the
developers
listen
to
the
operators
and
all
of
a
sudden
you'll
start
to
have
these
amazing
conversations
where
developers
will
trying
to
be.
You
know
they'll
try
to
write
some
algorithm
or
some
distributed
systems.
Problem
and
they'll.
Have
this
great
thing
and
it
works
great
in
theory,
then
you
would
operator
that
steps
in
and
says
so.
That's
great
but
you're
telling
me
that
you
need
40
gigs
a
second
for
network
traffic
and
our
data
center
is
running
cat,
5e
cable,
which
is
maxed
at
like
10
over
short
distances
and
five
in
reality.
A
So
that
thing
that
you're
trying
to
do
like
works
great
on
paper
and
works
great
on
your
local
laptop,
but
it
will
never
work
in
production
unless
we
upgrade
the
cabling
in
our
data
center
and
just
that
small
collaboration
saves
the
company
millions
or
even
billions
of
dollars
investing
in
a
software
project.
That
cannot
be
successful
because
the
underlying
hardware
won't
support
it,
and
this
is
just
one
example
of
how
putting
developers
and
operators
physically
in
the
same
space
helps
improve
the
software
delivery
cycle.
A
If
you
look
at
the
DevOps
manifesto,
there's
kind
of
five
key
categories
that
DevOps
is
broken
into,
the
first
is
reducing
organizational
silos
and
I've
already
talked
a
little
bit
about
this.
How
do
we
reduce
the
silos
that
exist
between
developers,
the
people
writing
code
and
operators,
the
people
making
sure
that
code
continues
to
run?
What
we
quickly
found,
though,
is
that
that
was
just
the
biggest
mountain
after
you
reduced
the
silos
between
developers
and
operators.
Who
knows
what
the
next
giant
mountain
is
security,
legal
review,
marketing
PR?
A
We
also
want
to
involve
the
security
team,
because
if
we
involve
the
security
team
early
instead
of
our
Security
and
Privacy
review
taking
six
months
for
someone
who
has
no
idea
what
our
product
is
was
no
understanding
of
our
core
business
goals.
We
instead
have
someone
who's
regularly
attending
our
stand-ups
they're,
regularly
contributing
code
in
or
reviewing
code,
and
then,
when
it
comes
time
to
actually
do
the
review,
they
may
be
able
to
complete
it
in
weeks
instead
of
months,
ultimately
delivering
software
faster,
but
also
safer,
because
they're
far
less
likely
to
miss
something
right.
A
The
second
piece
of
the
DevOps
manifesto
is
that
we
have
to
accept
failure
as
normal.
If
you
recall
earlier,
I
talked
about
fire
being
fired
a
lot
Clippers
worried
about
being
fired.
If
they
don't
ship
features,
operators
are
worried
about
being
fired
if
they
don't
deliver
100%
availability.
This
is
not
okay.
This
doesn't
create
a
culture
in
which
people
humans
can
thrive.
We
have
to
accept
failure
as
normal
any
system
that
humans
build
is
inherently
unreliable
and,
in
fact,
I
would
challenge
anyone
to
find
a
system
in
nature
that
is
100%.
Reliable.
A
A
So,
as
a
concrete
example,
let's
say
we're
doing
a
database
migration
we're
about
to
roll
out
a
database
migration
before
we
do
that,
let's
plan
for
failure,
let's
accept
that
failure
is
going
to
happen
so
before
we
actually
roll
out
that
database
migration,
we're
gonna
plan
a
rollback,
we're
gonna
write
a
script
that
rolls
back
our
deployment
and
rolls
back
the
changes
to
the
data
model.
Well,
why
would
we
invest
that
effort
in
advance?
It
might
just
work
and
you're
right.
A
It
may
totally
work,
and
that
was
wasted
effort,
but
the
problem
is:
if
it
fails.
If
that
deployment
or
that
database
migration
fails.
Now
your
phones
are
ringing,
your
pagers
were
going
off.
Social
media
is
blaring,
your
boss
is
yelling
at
you.
Your
boss's
boss
is
yelling
at
you.
The
site
is
down
you're,
losing
money
and
you're.
Trying
to
build
a
plan,
it's
not
the
best
time
to
build
a
plan.
The
best
time
to
build
a
plan
is
when
there's
not
a
lot
of
pressure,
and
you
can
think
clearly
so
by
accepting
failure
is
normal.
A
A
On
a
similar
note,
the
next
kind
of
piece
of
the
DevOps
movement
is
this
idea
of
implementing
gradual
change.
If
you
work
in
a
waterfall
software
development
methodology,
you
may
deploy
once
a
year
twice
per
year
and
the
problem
is,
if
you're
doing
that,
you're
deploying
hundreds
or
millions
of
lines
of
code
at
one
time
and
the
chance
that
there
are
no
bugs
in
those
million
lines
of
code
is
effectively
zero.
A
It
is
very,
very
small,
but
there
are
zero
bugs
in
a
million
lines
of
code
I,
don't
care
how
good
of
a
software
engineer
you
are,
how
rigorous
your
discipline
is,
there's
going
to
be
a
bug,
so
you
deploy
the
software
and
some
users
start
complaining.
Well
now
you
only
have
to
search
through
a
million
lines
of
code
to
find
the
bug.
It's
not
that
long.
It
might
take
another
year
versus
deploying
small
incremental
changes
if
we
deploy
say
ten
or
a
hundred
lines
of
code
at
a
time.
A
If
all
of
a
sudden,
our
monitoring
starts
failing,
our
users
are
yelling
at
us
on
social
media.
Our
boss
is
telling
us
that
something
is
broken
or
our
internal
users
are
saying
hey.
This
is
slow.
Now
we
know
exactly
where
to
look
for
the
problem.
The
smaller
that
change
the
easier
it
is
for
us
to
identify
the
problem
and
the
faster
it
is
for
us
to
fix
that
bug
or
roll
back.
The
change
rolling
back
a
million
lines
of
code
is
a
lot
harder
than
rolling
back
ten.
A
A
Oh,
we
deploy
six
thousand
four
hundred
28
times
per
day
and
I'm
like
great
I,
can
add
a
comma
to
some
Java,
six
thousand
four
hundred
eighty
six
times
per
day,
like
that's,
not
very
exciting
to
me,
but
the
frequency
at
which
you
deploy
relative
to
your
business
and
relative
to
your
industry
is
a
signal.
It
does
tell
you
how
how
well
you're
practicing
these
methodologies
if,
once
per
week
is
standard
for
your
industry,
maybe
you're
in
something
like
FinTech
or
banking
or
there's
regulatory
requirements,
and
and
that's
the
industry
standard,
that's
fine!
A
A
The
next
piece
of
the
DevOps
movement
is
this
idea
of
leveraging
tooling
and
automation,
and
this
is
where
you'll
often
see
people
confuse
tools
like
chef,
puppet
ansible,
salt
terraform
as
DevOps
tools,
and
that's
because
those
tools
largely
guided
the
DevOps
movement.
They
largely
supported
that
DevOps
movement
and
they
coincided
with
the
DevOps
movement.
So
a
lot
of
people
are
like
I'm,
a
DevOps
engineer
and
I'm
like
what
do
you
do
now,
like
I
write,
terraform
configs
like
okay,
maybe
not
the
appropriate
job
title.
A
That's
fine,
but
leverage,
tooling
and
automation
is
a
key
piece
of
the
DevOps
movement.
What
we
found
is
that,
after
breaking
down
those
barriers
and
after
accepting
failure
is
normal
and
implementing
gradual
change,
it
turns
out
there's
just
a
lot
of
work
like
creating
users,
installing
packages
building,
docker
containers,
monitoring
logging,
alerting,
like
all
of
those
things
take
time
and
if
you're
a
company
that
has
a
hundred
thousand
VMs
and
you
need
to
roll
out
an
open,
SSL
patch
you,
you
can't
have
people
sit
at
a
keyboard,
ssh
into
them
and
run
yum
update.
A
It
doesn't
scale,
and
we
quickly
learned
this
right.
We
quickly
learned
that
we
have
to
have
tooling.
We
have
to
have
a
dimension
in
order
to
successfully
implement
DevOps,
because
otherwise
you're
just
constantly
chasing
your
tail
you're
running
around
putting
out
fires
when
instead
we
need
to
be
leveraging
automation
and
tooling
to
make
things
repeatable
and
to
make
a
pattern
out
of
these.
A
Another
key
point
is
that
humans
are
inherently
very
bad
at
doing
the
same
thing.
Over
and
over
and
over
again
we
get
bored
we
get
distracted,
it
will
look
a
butterfly,
whereas
computers
are
really
good
at
doing
the
same
thing
over
and
over
and
over
and
over
again.
So
we
should
leverage
computers
for
doing
the
same
thing
over
and
over
and
over
again.
A
The
last
piece
of
the
DevOps
movement
is
that
we
have
to
measure
everything
it
doesn't
matter.
If
you
do
all
of
these
things,
if,
at
the
end
of
the
day
your
boss
comes
to
you
and
says
how
much
more
successful
is
the
business?
And
you
say,
people
are
happier
that
that's
not
a
business
justification
right
and
I'm,
not
saying
that
we
should
be
justifying
everything
we
do
with
money
or
deploys
per
day,
but
we
have
to
have
numbers
to
support
the
efforts
that
were
driving.
A
If,
today,
you
don't
have
any
metrics
and
you
implement
all
of
these
DevOps
things
and
all
and
everyone
feels
better,
and
you
know
what
the
business
is
better.
But
when
your
manager
comes
to
you
and
says,
hey,
we've
invested,
you
know
two
million
dollars
over
the
course
of
a
year.
We
hired
a
bunch
of
people.
We
bought
these
software
packages,
we
invested
in
this
tooling.
What
do
you
have
to
show
for
it?
And
you
don't
have
a
tangible
metric
to
be
able
to
point
to
you.
A
It's
unlikely
that
the
effort
will
continue
and
on
the
flip
side,
if
you
don't
measure
everything,
how
will
you
know
if
you're
actually
successful?
You
have
to
set
clear
metrics
for
success,
and
that
includes
at
the
organizational
level,
but
also
at
the
application
level.
Now,
there's
also
a
difference
between
measuring
everything,
monitoring,
everything
and
alerting
on
everything.
These
are
very
different
things.
You
may
measure
CPU
usage
memory
usage
available
disk
space.
A
A
Why?
We
need
to
measure
an
alert
on
things
that
matter
to
our
users.
Is
the
checkout
page
working?
Can
people
actually
buy
things
from
our
e-commerce
site?
Can
our
internal
users
run
their
analytics
and
their
reports?
That's
what
matters
that's
what
we
alert
on
then
we
measure
the
other
things
so
that
when
we
get
that
alert,
we
can
find
the
root
cause
or
the
root
causes
that
triggered
that
alert.
A
So
what
you'll
notice
here
is
that
these
are
all
abstract
ideas,
though
I
know
this
is
shocking.
You
no
idea
the
slide
was
coming.
These
are
all
abstract
ideas.
I
gave
you
some
examples
of
how
you
might
say,
reduce
organizational
silos.
Put
people
in
the
same
room.
I
gave
you
some
examples
of
measuring
versus
monitoring
versus
alerting,
but
there
are
abstract
ideas
right.
The
way
your
company
might
accept
failure
is
normal.
It's
very
different
than
the
way
another
company
might
accept.
A
Failure
is
normal,
and
what
we
found
is
that
a
lot
of
companies
don't
like
abstract
ideas.
They
want
more
concrete
implementations,
so
story
time
independently
of
DevOps.
Around
the
same
time,
though,
Google
started
this
discipline
called
sream,
which
stands
for
site
reliability,
engineering.
It
was
an
engineering
based
discipline,
so
kind
of
on
the
the
engineering
ladder
or
the
engineering
pay
scale.
A
If
you
will
of
people
who
build
and
maintain
reliable
systems,
their
job
was
to
keep
at
the
time
things
like
search
and
ads
up
and
running
all
of
the
time
or
as
much
of
the
time
as
possible.
So
sre
is
is
kind
of
like
a
very
prescriptive
way
to
do
DevOps,
and
this
is
why
you
might
hear
us
say
this
phrase
every
so
often,
which
is
that
class
sre
implements
DevOps.
Let
me
explain
what
I
mean
by
that,
so
sre
evolved
independently
from
DevOps.
A
It
was
kind
of
Google
was
really
in
its
own
little
bubble
at
that
time,
and
they
arrived
at
Sree
is
the
way
to
build
and
maintain
and
run
production
systems
at
scale
and
DevOps
was
kind
of
built
by
the
community
and
Google
and
SRT
was
kind
of
built
by
Google.
At
the
time.
Very
recently,
we've
learned
that
SOE
should
be
shared
with
the
world
for
quite
some
time.
Google
thought
that
SRA
was
like
our
secret
sauce.
If
you
will,
it
was
the
thing
that
differentiated
us
from
our
competitors.
A
It
was
the
thing
that
differentiated
us
from
other
cloud
providers.
We
shouldn't
talk
about
it,
it's
a
secret.
We
didn't
even
post
job
postings
for
it,
but
very
quickly.
We
learned
that
there's
a
language
to
sream,
there's
nomenclature,
there's
way
to
think
about
production
systems
that,
as
a
cloud
provider
when
we
go
to
a
customer-
and
we
start
saying
these
words-
we
start
talking
about
these
concepts.
They
have
no
idea
what
we're
talking
about.
So
we
go
to
a
customer
and
we
say
hey.
A
You
know
this
service
has
three
nines
of
availability
and
you're
depending
on
it,
which
means
you
can
never
never
have
more
than
three
nines
of
availability
and
our
customer
is
just
look
at
us.
Like
nines
question
mark
like
the
German
Shepherd
with
the
head
tilted
sideways,
and
this
is
when
we
decided
that
essary
doesn't
have
to
be
a
secret,
it
doesn't
have
to
be
a
secret
sauce
and
in
fact
other
companies
can
practice
SRE
and
then
Google
stepped
outside
of
its
bubble
and
realized
that
there's
already
a
practitioner
based
community
in
the
DevOps
community.
A
That
is
trying
to
do
this
they're
doing
it
successfully
in
some
cases,
unsuccessfully
in
other
cases,
and
that's
because
it's
not
prescriptive
DevOps
is
like
an
abstract
class
or
an
interface
in
programming.
It
says
here
are
the
things
you
should
do:
go
figure
it
out,
whereas
sre
is
a
concrete
implementation
of
that
class.
It
says
here's
how
you
reduce
organizational
silos.
Here's
how
you
accept
failure
as
normal
here
is,
how
you
measure
everything,
and
this
is
why
we
say
class
sre
implements
DevOps
now,
just
like
in
programming.
A
You
may
have
an
abstract
class
or
in
it,
an
implementation
of
an
abstract
class
that
has
other
methods
that
are
not
in
the
interface
right.
You
may
have
additional
methods,
additional
functions
that
aren't
in
the
interface
an
SRE
is
the
same
way.
There
are
things
in
the
SRA
discipline,
then
aren't
really
part
of
the
DevOps
interface,
but
sre
does
satisfy
the
DevOps
interface,
and
let
me
show
you
how.
So
if
we
go
back
to
the
previous
slide,
where
we
kind
of
talked
about
the
five
key
areas
of
DevOps,
how
does
sre
reduce
organizational
silos?
A
Well,
the
first
way
that
we
do
this
is
we
share
ownership
with
developers,
so
s
Ari's
at
Google
share
ownership
with
the
developers
by
using
the
same
shared
set
of
tooling
across
the
organization.
So
there's
a
single
set
of
tools
that
both
software
engineers
and
us
are
easy
use
for
production
systems
and
by
leveraging
the
same
set
of
tools
you
have
software
engineers
contributing
you
have
site,
reliability,
engineers
contributing
and
you
start
to
build
high
performance,
a
very
reliable
tooling
that
helps
people
get
their
job
done
on
the
other
side
of
the
organization.
A
How
does
SOA
accept
failure
as
normal?
Well,
one
of
the
ways
in
which
we
encourage
that
collaboration,
I
just
talked
about
is,
through
these
things
called
SLO
s,
service
level,
objectives
which
I'll
talk
about
more
in
detail
in
a
bit,
but
these
service
level
objectives,
in
addition
to
kind
of
forcing
collaboration
between
these
different
groups
in
the
organization.
A
They
also
force
us
to
admit
how
reliable
or
how
unreliable
our
system
can
be,
and
by
having
that
conversation
we
immediately
admit
that
our
service
is
going
to
have
faults
and
as
a
developer
and
as
a
product
owner
and
as
a
site,
reliability
engineer
I
can
determine
how
much
fault
I
actually
want
my
product
to
have
if
I'm
building
a
product
that
is,
you
know
up
against
some
competition,
it
needs
to
hit
the
market
very
quickly.
I
may
accept
more
risk
so
that
I
can
deploy
faster
I
can
deploy
riskier
changes,
etc.
A
If
I'm
building
a
product,
that
is
targeting,
say
the
healthcare
space
or
the
aviation
space
like
that
needs
more
nines
of
availability
that
has
to
work
all
of
the
time
we
can't
have
hours
and
hours
of
downtime.
We
have
to
have
redundancy
and
availability
and
that
may
slow
down
our
development
velocity,
but
this
is
a
conversation
that
occurs
between
the
product
owners,
the
developers
and
the
site
reliability
engineers.
A
Additionally,
as
you
might
expect,
the
essary
discipline
strongly
encourages
blameless
post-mortems,
which
are
popular
among
the
DevOps
community
as
well.
When
an
outage
occurs,
it's
nobody's
fault
and
instead
we
try
to
find
ways
in
which
we
can
improve
the
system
moving
forward,
something
that
sre
does
the
DevOps
doesn't
do
is
that
we
then
generate
metadata
about
our
post
mortems
globally
across
the
country,
our
country
company,
so,
for
example,
at
Google.
A
We
know
that,
like
the
vast
majority
of
our
outage
come
from
a
bad
configuration
change
and
that's
a
publicly
shared
statistic
and
the
reason
we
know
that
is
that,
in
addition
to
doing
an
isolated
post
mortem
about
an
incident,
we
then
categorize
and
catalog
all
of
that
information
into
a
database
where
we
can
run
reports
and
gain
statistics
about
what
is
causing
these
overall
outages
and
that
can
help
you
build
kind
of
meta,
tooling
and
and
better
improve
the
system.
Overall.
A
A
Tolling,
an
automation
is
really
a
key
piece
of
the
SRA
culture.
In
fact,
I
sorry's
have
this
thing
called
toil.
Toi
l
I'll
talk
about
it
more
in
a
little
bit,
but
toil.
Is
this
idea
that
you
should
you
should
be
spending
on
times
that
bring
long-term
value
to
the
system?
So
you
should
be
investing
your
time
and
investing
your
resources
and
things
that
bring
value
to
the
system
in
the
long
run.
Ssh
into
a
system
and
restarting
a
service
doesn't
bring
long-term
value.
A
It
brings
immediate
term
value,
but
that's
something
that
we
should
fix
in
the
long
term,
so
sres
invest
heavily
in
tooling
and
automation
in
order
to
automate
this
year's
java
way.
That's
kind
of
an
unofficial
tagline
among
a
lot
of
us
Ari's
at
Google
is
that
the
manual
tasks
that
I
did
this
year,
I
shouldn't
have
to
do
again
next
year.
I
should
automate
those
away.
A
A
Sree
is
a
very
prescriptive,
very
concrete
way
to
satisfy
the
DevOps
interface.
So,
given
that,
let's
jump
into
a
few
key
areas
that
I
think
are
important
to
discuss
the
first
who's
SL
is
SL
OHS
and
SLA
is
oh.
My
an
SOI
is
a
service
level
indicator,
something
like
request,
latency
or
requests
per
second
or
failures
per
request.
There
a
point
in
time
or
an
aggregate
point
in
time
of
a
particular
metric
about
a
system.
A
Basically,
an
SLI
tells
you
at
any
given
moment
yes
or
no
for
a
metric
in
a
system.
Is
it
healthy
or
is
it
not,
and
you
define
that
right
so
for
your
system,
healthy
may
be
that
the
ratio
of
successful
requests
to
failed
requests
is
less
than
1%
right,
so
99%
of
your
requests
are
successful.
So
at
any
point
in
time,
if
you
imagine
a
point
in
time,
that's
a
binary
operation,
yes
or
no
at
this
moment
in
time,
are
we
up
or
down
per
that
metric?
A
This
is
a
business
agreement
that
happens
between
a
customer
or
a
consumer
and
a
service
provider.
The
SLA
is
typically
based
on
the
SL
o.
Ideally
you
want
to
break
your
own
internal
SLO
before
you
break
an
external
SLA,
because
violating
an
SLA
typically
means
you
have
to
give
money
or
credits
or
reparations
in
exchange
for
violating
that
SLA.
A
A
But
to
give
you
a
better
understanding
of
kind
of
who
is
involved
in
this
process,
I
built
this
handy
slide
so
product
s,
re
and
software
engineering
work
together
to
build
those
SL
is
up
or
down.
Then
the
SR
reason
the
product
worked
directly
to
determine
what
that
looks
like
over
a
period
of
time
and
again
this
is
a
function
of
how
fast
do
we
want
to
move?
What
are
we
up
against
in
the
market?
How
much
risk
are
we
willing
to
accept?
A
What's
our
target
market
and
then
SLA
is
are
generally
built
by
the
sales
teams
and
the
customers,
maybe
with
some
negotiations
from
product,
but
they
have
to
be
tighter
or
or
less
than
the
SLO,
because
again
you
want
your
SLO
to
break
before
your
SLA
breaks,
and
sometimes
s
willies
are
also
used
as
part
of
a
sales
engagement
right
where
you
can
like
buy
more
availability.
So
that's
not
really
part
of
the
s.
Re
conversation
other
than
the
SLO
informs
kind
of
the
minimum
baseline
for
the
SLA.
A
So
I
have
to
get
this
question,
which
is
like
okay,
so
you
you
have
this
SLI
thing
and
then
this
SLO
thing
and
then
like
what
happens
when
you
like
go
over
like
okay,
my
system
needs
to
be
99.99%
reliable.
That
gives
me
like
twelve
and
a
half
minutes
of
downtime
per
quarter.
What
happens
when
I'm
out
like
can
I
just
keep
pushing
do
I
fire,
my
developers
like
what
happens
next
well,
this
is
where
error
budgets
come
in.
A
So
one
of
the
things
that's
important
to
note
is
that
it
is
nearly
impossible
to
find
a
system
that
is
100%
reliable
and
it
is
often
cost
prohibitive
to
build
a
system
that
is
100%
reliable,
especially
when
you're
relying
on
third-party
components
take.
For
example,
this
two
dimensional
material
phone
on
the
screen
this
screen
or
this
phone
has
99%
reliable
with
its
cellular
network
carrier.
This
is
actually
pretty
standard
in
most
countries.
A
A
The
user
would
only
experience
your
app
at
99
percent
availability
because
they
are
governed
by
the
least
reliable
components
of
their
system.
So
even
if
you
have
100%
connectivity
between
your
data,
centers
and
the
cellular
network
towers
that
user.
That
end-user
will
only
experience
99
percent
availability
because
they
are
governed
by
the
least
reliable
component
in
the
system-
and
this
is
a
key
point-
is
that
optimizing
for
100%
availability
isn't
just
difficult
oftentimes,
it's
irresponsible!
It
is
not
in
your
company's
best
interest
to
optimize
for
100%
availability.
A
Instead,
you
should
be
accepting
failure
as
normal
and
understanding
how
much
failure
is
available
for
your
system,
so,
like
I,
said
that
user
only
experiences
99%
availability.
So
how
do
you
determine
how
much
risk
your
service
can
tolerate?
How
do
I
know
how
risky
my
service
can
be?
Well,
there
are
many
factors
to
consider
like
fault,
tolerance,
availability,
competition
in
the
market,
how
fast
you're
trying
to
deliver
whether
there's
a
giant
conference
that
you
need
to
launch
at
that
was
a
joke,
but
you're
acceptable
risk
dictates
your
SLO.
A
So
if
you
have
a
product
that
has
really
critical
market
timing-
and
you
know
that
you
need
to
deliver
features
quickly,
you
may
say:
hey
we're.
Only
gonna
offer
one
nine.
Ninety
percent
availability,
because
we
need
to
be
able
to
move
quickly
and
we
don't
want
to
have
to
focus
on
reliability
right
now.
We
want
to
focus
on
shipping
new
product
and
new
features,
but
again,
if
you're
in
a
different
industry
like
health
care
or
aviation,
where
reliability
is
super-important,
you
may
say
like
hey
look.
We
only
want
to
focus
on
reliability
right
now.
A
We
need
to
improve
our
nines
of
availability
so
that
our
customers
can
trust
us.
So
as
long
as
your
SLO
s
are
met,
you
can
continue
pushing
new
features
and
new
product.
But
what
happens
if
you
violate
that
SLO?
What
happens
if
you've
exceeded
your
error
budget,
which
is
the
amount
of
failure
you
can
have
within
your
SLO?
Well
then,
everyone
just
like
plays
ping-pong
right.
That's
how
this
works.
No,
you
can
continue
deploying
your
developers
can
continue
building
features,
but
everything
has
to
focus
on
reliability.
A
A
So
just
like
a
bank
account
where
you
get
a
paycheck
and
the
money
goes
in
and
then
you
buy
some
pokemon
cards
and
then
the
money
goes
out
and
and
then
you
get
more
money
right.
The
error
budget
works.
The
same
way
is
that
after
we've
exhausted
our
bank
account
of
error,
we
can
only
focus
on
reliability.
We
can
only
deploy
features
that
focus
on
reliability
or
bugs
that
improve
reliability.
A
So
what
does
that
actually
look
like?
Well,
here's
kind
of
a
prettied
up
version
of
an
internal
graph,
so
on
the
top
we
have
the
health
of
the
system,
which
is
measuring
the
number
of
requests
that
are
under
300
milliseconds
at
any
point
in
time.
So
this
is
like
an
sli
in
the
middle.
In
the
green
we
have
the
compliance
that
up
or
down
are.
A
Then
at
the
very
bottom
we
have
the
budget
of
non-compliant
requests
in
our
SLO
budget.
So
how
many
requests
are
remaining
well?
You'll
notice,
there
is,
is
pretty
far
on
the
right
hand,
side,
there's
a
drastic
increase
in
our
latency,
so
you'll
notice
the
little
bit
confusing
because
the
chart
goes
down,
which
is
a
decrease
in
the
number
of
requests
under
300
milliseconds,
which
is
an
increase
in
the
number
of
requests
over
300
milliseconds
right.
A
We
have
to
focus
on
reliability,
so
you
might
ask
yourself:
well
what
just
prevents
developers
from
saying
well
like
I
know
I'm
at
the
budget,
but
like
this
is
an
important
feature.
Why
can't
I
deploy
it?
Well,
they
can,
but
they
might
lose
their
SOE
support.
So
remember
that
sre
is
a
discipline.
It's
a
separate
organization
that
partners
with
the
software
and
product
teams
and
if
the
product
team
and
the
software
team
are
not
willing
to
be
adequate
partners.
A
A
A
Toil
is
best
described
as
what
it's
not
those
weird
things
toil
is
like
not
email.
It's
not
expense
reports,
it's
not
meetings,
it's
not
traveling!
These
are
all
things.
We
call
overhead
right
things
that
are
required
to
do
your
job,
that
pretty
much
everyone
in
the
organization
needs
to
accomplish
toil
is
actually
something
that
is
manual
repetitive.
Most
importantly,
it's
devoid
of
long-term
value.
A
It's
often
incredibly
tactical
and
highly
automated
classic
example
of
toil-
is
like
SS
aging
into
a
system
and
restarting
a
service,
because
it's
out
of
memory
or
it's
out
of
you
know
spiking
CPU
usage
or
something
crazy
right.
That's
incredibly
tactical
like
right
now
doing
that
graphs
go
back
down
everyone's
happy,
but
it
is
devoid
of
long-term
value
because
that
service
is
likely
going
to
out
of
memory
again
and
you're.
Gonna
have
to
do
it
again
and
it's
gonna
be
repetitive
and
it's
manual,
but
it
is
very
easy
to
automate
right.
A
That's
a
classic
example
of
something
that's
very
tightly
and
in
the
SRA
discipline
we
measure
toil
and
we
talk
about
toil
because
it's
a
very
negative
consequence
of
the
job.
If
you're
constantly
working
with
toil,
this
can
lead
to
like
career
stagnation.
No
one
got
promoted
for
restarting
servers
right,
how
many
people
got
promoted
because
they
restarted
a
server
once
right,
exactly
no
one
and
at
the
same
time
a
little
bit
of
toil
is
also
good.
A
So
there's
a
careful
balance
here
and
every
year
we
kind
of
send
out
a
survey
to
all
of
the
authorities
at
Google
and
most
us
our
ease
aim
somewhere
between
10
to
20
percent
of
toil
and
their
job.
So
you
might
ask
yourself,
where's
toil
good.
It
sounds
like
this
is
terrible.
I
should
automate
everything.
A
Well,
if,
once
per
year,
you
have
to
do
some
very
complex
operation
like
some
aggregate
report
that
spans
multiple
systems,
it's
very
complex
and
it
would
take
you
20
hours
to
automate
it,
but
it
just
takes
you
15
minutes
to
do
it
manually,
and
you
only
need
to
do
it
once
per
year.
It
is
not
a
good
return
on
investment
for
you
to
spend
time
automating
that
instead,
you
should
make
sure
that
other
people
on
the
team
know
how
to
do
it
so
that
you're,
not
one
person,
documented
etc.
A
But
you
shouldn't
waste
more
time,
automating
something
then
it
takes
to
do
it
over
time.
It
would
take
a
hundred
years
for
you
to
reclaim
that
time.
No,
we
take
you
125
years
to
reclaim
that
time.
It's
just
not
worth
it
toil,
is
also
an
excellent
way
for
newcomers,
like
interns
or
people
new
to
the
team
to
learn
the
system.
A
There's
no
I
mean
there
are
better
ways,
but
one
of
the
best
ways
to
learn
a
production
system
is
to
explore
it
right
and
being
able
to
poke
around
and
understand
systems
and
understand
how
they
work
with
one
another.
That's
very
toi
Lee,
but
it's
also
a
great
learning
opportunity
for
newcomers
to
the
team.
A
Another
value
for
toil
is
that
how
many
people
have
ever
had
a
bad
day?
Wow.
That's
like
you
were
also
positive
people
right.
Okay,
how
many
people
have
ever
had
it
straight
like
6
hour
a
meeting
day?
I
know
that
you've
all
had
that
yeah
right
and
then
at
the
end
of
6
hours
you
have
475
unread
emails.
You've
basically
feel
like
you've
accomplished
nothing
for
the
day
and
then
you
get
a
page
they're
like
this
servers
out
of
disk
space,
and
you
can
fix
it.
A
Then
you
can
go
home
for
the
day
and
you
feel
like
you've
done
something
with
your
life
right
toil
can
actually
satisfy
that
right
toil
provides
that
instant
gratification,
but
in
small
doses,
at
a
large
grand
scale.
We
want
to
eliminate
toil
as
much
as
possible.
We
want
sres
to
be
focused
on
improving
the
reliability
of
the
system
and
the
availability
of
the
system
not
performing
toilet
asks
unless
absolutely
necessary.
So
I've.
A
Given
you
kind
of
a
brief
overview,
you
may
ask
yourself:
where
can
I
learn
more
there's
this
happy
link
at
google.com,
slash
sre,
where
you
can
find
all
of
this
information
and
a
bunch
more
including
two
free
books?
That's
free
ebooks.
The
first
is
the
site,
reliability,
engineering
book
I
like
to
call
this
the
theoretical
calculus
book
and
then
there's
the
site.
Reliability
workbook,
which
is
like
the
the
problem
set.
If
you
will
they're
both
excellent
books,
the
sre
book
talks
a
little
bit
more
about
the
theory
and
the
history.
A
The
workbook
is
a
little
bit
more
practical,
a
little
bit
more
hands-on
and
there's
an
entire
section
about
how
DevOps
and
SOA
relate
to
one
another.
Both
of
these
are
free
did
I
mention
that
they're
free
and
you
can
download
them
for
free.
If
you
would
like
a
print
version,
you
can
purchase
them
from
your
favorite
book
vendor.
A
So
to
kind
of
conclude
here
is
sre
DevOps
2.0?
No,
it's
not
trying
to
be
I,
also
hate
when
people
put
2.0
on
things
is
this:
are
you
trying
to
overtake
DevOps
no
Lowell
and
I
learned
yesterday
that
I'm,
the
only
person
who
pronounces
Lowell
I
thought
people
pronounced
it?
Mul
can
I
adopt
both
DevOps
and
sre.
Yes,
as
I
said
before
right,
if
you
implement
DevOps,
if
you
follow
the
essary
workbook
and
and
that's
our
ebook,
you
will
be
satisfying
the
DevOps
interface
on
the
flip
side.
A
If
you
satisfy
the
DevOps
interface,
you
might
not
be
doing
SRE.
There
are
a
number
of
other
disciplines
like
Victor.
Ops
has
a
thing
called
prod
ops
and
there
are
other
disciplines
that
satisfy
the
DevOps
interface,
but
are
not
sre
its
DevOps
dead.
I
get
this
question
a
lot
like.
No
one
really
talks
about
DevOps
anymore,
I.
Think
that's
because
it's
not
dead.
It's
just
assumed
now
right.
It's
assumed
that
you're
following
some
or
all
of
these
principles.
A
As
my
talk
over
yes,
we
have
about
five
minutes
for
questions.
There
are
two
microphones
in
the
middle
of
the
room.
If
anyone
has
questions,
please
line
up
at
the
microphone
for
the
recording,
while
folks
are
lining
up.
If
you
have
a
question
that
you
don't
feel
comfortable
asking
on
the
microphone
or
on
the
recording,
that
is,
my
twitter
handle
feel
free
to
tweet
at
me
at
any
time.
My
direct
messages
are
also
open.