►
Description
Designing for Reliability in Production - Ayelet Sachto, Google
Speakers: Ayelet Sachto
Learn the considerations and strategies that can make designing for reliability in production more intentional; going beyond infrastructure into operational practices and best practices for application design and operational readiness when designing an application for production. *Sections*: -Why & what of reliability -Building reliable systems - what to think about when designing reliable systems and some examples from the field -Running reliable systems - How we leverage SRE and how we can run reliable systems using SRE principles, as well as lessons from the field.
For more Continuous Delivery Foundation content, check out our blog: https://cd.foundation/blog/
A
A
Why
it's
important
and
I'll
introduce
a
few
best
practices
for
designing
and
running
reliable
production
systems.
At
the
end,
I
will
also
share
with
you
how
we
can
leverage
sre
and
share
with
you
some
resources
to
get
started,
why
we
are
focusing
on
reliability,
why
it's
important
that
our
service
will
be
reliable,
put
yourself
in
the
user
shoes
for
a
second,
let's
say
you
are
using
an
application
with
great
features,
but
you
can
trust
it
to
work.
A
Would
you
care
if
an
application
has
an
amazing,
ui
and
features
if
it
just
doesn't
work
people
sometimes
replace
reliability
with
availability,
but
reliability
is
more
than
just
service
availability,
especially
when
we're
talking
about
distributed
complex
systems.
The
definition
of
reliability
is
more
complex
if
the
service
is
down
and
if
we
are
getting
a
response.
A
Think
about
it
does
reliability
means
the
same
for
netflix,
user
and
google
search
user
depends
on
your
service.
Your
users
expectations
will
be
different
and
reliability
can
mean
different
things
as
a
rule
of
thumbs.
A
system
is
more
reliable
if
it
has
fewer,
shorter
or
smaller
out
diseases.
You
probably
heard
of
a
three
nine
or
four
nine
service,
but
it
all
comes
down
to
the
amount
of
downtime
that
our
users
are
willing
to
tolerate.
Here
are
a
few
answers
that
our
customers
gave
about
reliability
and
what
it
means
to
them.
A
You
can
see
that
it
all
comes
down
to
the
customer,
the
user,
trust
and
satisfaction.
We
want
to
take
a
customer-centric
approach
because
unhappy
customers
will
leave
so
if
reliability
is
defined
by
our
users
to
understand.
What's
the
level
of
reliability
we
need
to
provide,
we
need
to
understand
the
user's
expectations
and
measure
them
as
closely
as
possible
to
the
user
expectations
and
experience
and
later
I'll
expand
about
slos
in
that
regard.
A
A
A
The
third
topic
is
data
handling
or
more
specifically,
avoiding
keeping
state
and
using
caching
and
charting
for
frequent
access
data.
So,
let's
zoom
on
each
one
of
those
one
of
the
fundamental
architectural
ways
in
which
we
can
improve
our
resilience
and
our
reliability
is
to
build
systems
spanning
across
zones
and
regions.
A
Designing
systems
that
span
these
zones
and
regions
or
choosing
products
that
do
do
so
automatically
allow
us
to
minimize
the
impact
and
provide
some
level
of
service
using
slas
as
reliability
floors.
We
can
assure
reasonable
expectation
of
reliability
for
our
applications.
Architectures
using
this
construct,
extending
to
multi-zonal
or
regional,
will,
provide
us
a
higher
resilience
and
will
lead
us
to
general
expectation
of
four
nights.
A
So
I
mentioned
that
failure
is
not
a
question
of
if
but
a
question
of
when
so
we
want
to
design
our
system
to
be
resilient,
so
it
is
important
to
understand
how
do
we
deal
with
failures
and
to
what
type
of
failures
we
are
resilience
to?
Can
we
tolerate
unexpected
single,
instant
failure
or
a
single
instant
reboot?
A
Our
system
will
deal
with
zonal
original
failures,
so
a
good
approach
to
mitigate
the
first
examples
of
an
instant
failure
or
reboot
will
be
to
use
persistent
disks
and
startup
scripts
to
quickly
reconfigure
software
and,
of
course
we
want
to
backup
our
data
to
mitigate
zonal
or
regional
failures.
We
will
want
to
have
resources,
diversity
across
region
and
zones
and,
of
course,
implement
load
balancing
in
front
of
them.
A
A
A
A
A
It's
important
to
note
also
that
some
data
is
accessed
more
frequently
than
others
and
for
performance
optimization.
We
might
want
to
use
sharding
and
caching
with
caching.
We
are
referring
to
partitioning
your
data
on
something
in
the
request,
for
example,
user
id
into
n
buckets
and
with
caching
I'm
referring
to
moving
the
data
access
to
faster
io
to
handle
more
frequent
data
access,
for
example,
run
we
want
to
do
it
in
order
to
reduce
the
overload
on
our
servers
and
eliminate
any
possible
bottlenecks
due
to
io.
A
A
A
In
that
perspective,
it's
important
to
choose
the
right
building
blocks
for
us
in
our
architecture
and
our
design.
When
choosing
the
right
building
blocks,
we
want
to
choose
a
gcp
product
that
suits
the
level
of
reliability.
We
actually
need
be
mindful
also
of
gcp
building
block
scope,
as
I
mentioned,
zonal,
regional
and
global,
and
that
those
correlate
with
our
application
targets
when
talking
about
design
and
architecture.
A
There
is
no
one-size-fits-all.
We
need
to
let
our
requirements
guide
us.
Let's
look
at
an
example
of
a
customer
facing
web
service
design
and
I'm
specifically
saying
an
example,
because
this
is
what
worked
in
our
case,
but
designs
will
be
very
different,
depends
on
the
scope,
criticality
and
traffic,
to
name
a
few
for
our
specific
use
case.
A
We
also
chose
gke
for
horizontal
scale,
both
for
the
front
end
and
back
end
pods,
as
it
can
accommodate
for
the
25x
increase
in
web
traffic,
with
a
combination
of
gclb
for
load
balancing
as
we
needed
global
solution
and
redundancy.
We
used
multi-region
gke
clusters
gclb,
as
mentioned,
and
three
availability
zones
to
ensure
reliability.
A
It's
also
important
that
we'll
have
a
robust
cicd
pipeline
for
that
we
used
the
cicd
stack
for
source
control
management,
continuous
integration
and
delivery,
artifact
management
and
software
supply
chains.
We
also
used
terraform
to
automate
the
deployment
and
provisioning
of
cloud
resources
using
configuration
file,
and
we
try
to
use
automation
as
much
as
possible
to
eliminate
to
it.
In
addition
to
increase
the
reliability,
we
set
up
canary
testing
for
new
releases
and
configurations.
A
We
mentioned
that
reliability
is
the
most
important
feature,
but
reliability
is
not
enough
and
we
need
to
balance
velocity
and
reliability.
Google
framework
for
that
is
sre.
Implementing
sre
includes
both
technical
and
cultural
practices.
We
want
to
use
data
to
guide
our
decisions,
to
foster
a
learning
culture
and
empower
our
teams.
A
A
We
assume
some
level
of
unreliability
that
our
users
are
willing
to
tolerate
that
amount
of
unreliability.
Is
our
budget
error
budget
create
a
common
incentive
for
business
operation
and
development
to
balance
velocity
and
reliability?
If
we
have
more
budget,
we
prioritize
velocity,
we
move
faster,
make
changes,
take
risks.
A
A
But
where
do
we
draw
the
line
between
happy
user
and
honep
user?
We
need
to
ask
ourselves,
in
which
point
our
user
will
notice
that
things
are
down.
We
call
that
measurement
a
service
level
indicator.
An
sli
tells
you
at
any
moment
in
time.
How
well
your
service
is
doing.
Is
it
do
it
acceptably
or
not?
A
A
It
defines
what
you
are
willing
to
do,
for
example,
money
refund.
If
you
are
failing
to
meet
your
objective
to
be
able
to
achieve
that,
we
need
our
slos
our
target
to
be
tighter
than
our
slas
by
that
slos
and
narrow
budget
align
incentive
across
the
product
lifecycle
and
help
us
to
make
data-driven
decisions.
A
A
journey
to
sre
includes
both
implementing
sre
practices
that
we
covered
before,
like
slos,
but
also
fostering
and
enabling
culture.
Accepting
failure
as
normal,
reducing
organizational
silos
and
making
tomorrow
better
than
today,
by
reducing
the
cost
of
failures
using
canary
deployments
and
having
ci
cd
pipeline
and
automation
in
general.
A
Finally,
in
this
link,
you
can
find
a
collection
of
publicly
available
resources
for
you
to
use
break
by
resource
type
and
level,
including
knowledge
d,
and
because
I'm
all
about
tata
and
feedback
is
an
amazing
gift.
I
really
appreciate,
if
you
take
a
few
minutes
from
your
time,
to
share
with
me
your
feedback
about
this
talk,
and
with
that
I
would
like
to
thank
you
for
your
time
and,
let's
move
to
questions.