►
Description
Join us for Kubernetes Forums Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Reliable Observability at Scale: Error Budgets for 1,000+ - Fred Moyer, Zendesk
"Observability and reliability engineering have been on a convergent course for several years. Error Budgets joined the reliability lexicon of engineering organizations in 2016 with the release of the SRE book. The intersection of observability and reliability has largely been the domain of specialists for practical implementation. How can one democratize these techniques to put them in the hands of a thousand engineers at once?
At Zendesk we developed simple algorithms and practical approaches for implementing SLIs, SLOs, and Error Budgets at scale using a number of observability tools. This talk will show the approaches developed and how we were able to manage observability instrumentation across dozens of teams quickly in a complex ecosystem (CDN, UI, middleware, backend, queues, dbs, queues, etc)."
A
Hey
folks
welcome
to
reliable
observer
ability
at
scale
our
budgets
for
1,000,
plus
I'm,
Fred
and
I
work
at
Zendesk
as
the
necessary.
We
did
a
lot
of
hiring
in
2019,
including
myself,
and
we
currently
have
around
1,000
engineers,
but
we
have
a
small
SRE
team.
We've
got
about
10
s
eries
there
and
we've
grown
from
about
one
right
around
a
year
ago.
So
essentially
we
have
one
s
re
for
every
hundred
engineers
and
for
comparison.
A
If
you
read
the
google
book,
they'll
say
that
they
have
one
s
re
for
every
ten
engineers
they
call
s.
Re
is
a
scarce
resource.
They
take
an
approach
of
embedding
sres
in
product
engineering
teams,
but
that's
not
something
that
we
could
do
because
of
our
limited
s,
eries
so
Zendesk.
We
consider
reliability
to
be
a
core
competency
of
ours
and
a
top-level
feature.
A
Since
we
do
produce
customer
support
software,
it
needs
to
be
up
all
the
time
and
it
needs
to
be
very
reliable
and
in
2019
we
embarked
on
an
engineering
wide
initiative
to
implement
product
level
era
budgets
across
geographically
distributed
teams
and
services.
This
appeared
to
be
a
daunting
challenge.
As
era
budgets
as
a
concept
in
the
relative
reliability,
space,
we're
still
kind
of
evolving
I
joined
Zendesk
in
June
of
this
year
and
I
kind
of
got
thrown
into
the
deep
end
of
this
and
folks
said:
hey,
you
know.
You
just
joined
your
era.
A
Budget
expert
I'm
like
great
so
hi,
I'm,
Fred,
Hama
I,
tend
to
think
of
myself
as
a
slow
jiton,
which
is
like
a
statistician,
but
not
quite
as
cool
I've
done.
A
lot
of
hacking
in
the
observability
space
I've
hacked
on
time-series
databases,
staff
see
from
atheists
particular
histograms
I've
written
a
lot
of
stuff
about
those.
Previously
I
was
with
a
monitoring
company
called
circ
onus
and
high
scale
web
service
called
Turnitin,
where
I
worked
on
a
large
lot
of
production
systems
broke
stuff
and
measured
stuff
and
I've
been
writing
code
for
a
while.
A
A
You
know
all
lug
learn:
grow
giant
monolith
that
everyone
depends
on
and
makes
a
lot
of
money,
but
as
as
things
grew,
we
started
to
implement
micro
services,
maybe
not
micro
services,
as
we
know
today.
But,
essentially
you
know
specialized
services
and
sort
of
a
service,
oriented
architecture
approach,
and
you
know
other
rails
apps
started
to
grow
and
we
started
to
acquire
other
companies.
The
chat
product
in
particular
is
based
on
pythons
twisted
web
server.
So
it's
a
markedly
different
approach
than
the
process
based
rails
server.
A
You
know
see
the
architectures,
you
know
already
started
to
get
a
bit
dissimilar
and
we
grew
more
rails.
Apps
added
some
services
in
Scala
go
and
also
add,
you
know
a
few
years
ago
started
to
move
things
into
kubernetes
on
AWS
and
we
added
more
apps
and
bought
more
companies.
In
this
case
the
new
newest
stolen
was
sell
about
a
year
ago
and
it's
something
like
if
you've
ever
worked
in
a
large
organization.
You
know
you'll
notice,
it's
not
a
perfect
cookie,
cookie
cutter
architecture,
you're
gonna
have
teams
and
products
that
follow
Conway's
law.
A
You
know
it
doesn't
make
sense
to
say
for
a
chat
application
to
implement
that
in
rails.
That's
just
not
the
right
solution
for
that
problem,
so
coming
to
a
unified
architecture,
is
something
that's
really
not
realistic
in
a
lot
of
cases,
especially
under
a
large
organ
engineering
organization.
But
with
that
in
mind,
what
approach
should
we
take
for
implementing
error
budgets
when
we've
got
kind
of
a
heterogenous
architecture
like
this?
A
So
let's
dive
into
that?
So
let's
first,
let's
take
a
look
at
a
little
bit
of
history.
You
know
we
kind
of
had
a
Cambrian
explosion
of
SLI,
SSL
OS
and
err
budgets
in
2016
with
the
Google
book
and
over
the
next
couple
years.
You
know
we
had
the
sre
reliability
workbook
come
out.
The
famous
video
losses
out
there
casting
right
now
by
Liz
and
Seth
about
SLO
as
SLA
as
SLI
is.
A
Oh
my
and
then
you
know
just
a
number
of
additional
talks
and
in
doing
research
for
figuring
out
how
we
should
democratize
our
budgets
across
the
you
know,
a
thousand
person
engineering
team
I
started
to
dig
into
the
definitions
of
SL
eyes
and
SLO
as
an
era.
Budgetary
closely
and
what
I
noticed
was
that,
in
you
know
the
sre
book.
A
You
know
you
had
certain
definitions
which
very
very
subtly
different
from
the
SOE
workbook,
which
were
subtly
different
from
other
parts
and
to
try
and
get
those
into
code
for
a
lot
of
people
and
also
in
a
format
that
you
can
plug
into
a
monitoring
system
became
an
exercise
where
I
had
to
be
very
pedantic
about
defining
exactly
what
these
work.
So
here's
what
I
came
up
with
so
SLI
is.
We
all
know
these
delineate
good
requests
from
bad
requests.
A
Here's
three
examples:
we've
got
a
percentile
based
latency
SLI.
On
top
there
we've
got
an
availability
based
SLI
in
the
middle,
and
we've
got
just
a
account
based
SLI
on
the
bottom,
and
these
are
all
English
statements
that
you
know
we
might
spout
off.
But
how
do
we?
You
know?
How
do
we
parse
these?
How
do
we
make
it
easy
to
understand?
A
A
It's
got
a
metric
operator,
you
know
less
than
not
equal
to
less
than
and
it's
got
a
metric
value,
and
so
you
put
all
three
of
these
together
and
you've
essentially
got.
You
know,
formula
that
you
can
throw
an
SLI
at
that
you
come
up
with
it
and
see
if
it
parses
well
and
if
you
can't
find
each
of
these
three
parts
in
your
SLI
chances
are.
A
Something
is
wrong
and
you
know,
depending
on,
if
you
have
a
bespoke
service,
you
might
choose
an
SLI
based
on
a
metric
for
that
service,
so
you
might
be
missing
one
or
more
of
these
s
ellos.
These
are
binding
targets
for
SLI
s
and
in
a
in
an
event-based
SLO,
you
essentially
have
the
number
of
good
requests
divided
by
the
number
of
total
requests
over
a
time
range.
A
So
let's
take
a
look
at
what
those
are
now
the
top
right,
one
with
percentiles,
that's
very
wordy
and
as
dave
chappelle
is
fond
of
saying,
that
stuff
is
confusing.
So,
let's,
let's
read
it
out.
99
percent
of
95th
percentile
homepage
latency
over
five
minutes,
less
than
500
milliseconds
over
the
trailing
month.
Wow,
that's
a
tough
one.
The
availability
one
is
easier.
You
know,
99
percent
of
my
requests
shouldn't
be
5x
x
over
the
past
7
days,
and
the
bottom
one
is,
is
a
lot
simpler.
A
95%
of
my
homepage
requests
should
be
under
100
milliseconds
in
the
last
24
hours.
How
do
we
parse
these?
Well?
We've
got
a
success
objective,
which
is
essentially
what
percentage
of
your
requests
need
to
meet
this
SLI
definition
of
good
requests.
We've
got
our
SLI
in
the
middle
and
we've
got
a
time
range.
The
time
range
is
the
one
that
I
see
left
off
in
most
cases
where
folks
don't
have
a
well
form
to
sli,
they'll,
say:
hey,
we
want
you
know,
95%
of
our.
You
know,
search
page
requests
to
be
under
100
milliseconds.
A
What
time
range
are
you
talking
about?
Are
you
talking
about
a
week?
Are
you
talking
about
an
hour
because
the
way
that
you
defined
this
time
range
tells
you
a
lot
about
who
the
audience
of
this
SLO
and
consequently
the
era
budget
is,
and
you
know
how
you
think
your
service
can
perform
there,
so
you
know
to
wrap
up
on
SLO
s
again.
A
You
know
you
have
to
have
these
three
parts
of
success:
objective
and
SLI
in
a
period
era
budgets
you
know,
nobody's
perfect,
you
know
we
have
our
budgets
so
that
we
can
you
know
we
don't
want
our
service
to
be
too
perfect,
because
then
we'd
never
be
able
to
innovate.
We
don't
want
to
move
fast
and
break
things,
but
we
do
want
to
move
fast
and
break
things
that
we
can
measure.
So
an
air
budget
is
essentially
one
your
SLO,
you
know
in
this
case
you
know
if
we
have
a
success
objective
of
99
percent.
A
A
Instead
of
a
success
objective
here,
I
have
one
my
success:
objective,
which
I'll
call
the
error.
Budget
I'll
have
my
SLI
and
then
I'll
have
my
period
same
type
of
formula,
and
this
is
something
we
gave
to
every
engineer
to
say,
like
you
know,
okay,
we're
in
this
company-wide
effort
to
implement
our
budgets.
You
need
to
have
something
that
looks
like
this
to
move
forward,
and
this
worked
remarkably
well
because
it
gave
them.
A
So
to
kind
of
sum
up,
the
you
know
keys
to
air
budget
democratization.
You
know
we
want
real-world
examples
that
are
easy
to
reference.
We
want
formulas
that
both
humans
and
code
and
monitoring
systems
can
process,
and
we
want
to
be
explicit.
We
want
to
be
pedantic
these
really
small
details
such
as
you
know,
the
fact
that
you
need
a
period
there.
A
time
range
can
make
a
really
big
difference.
So.
A
We
were
processing
Logs
with
elk
for
a
while,
but
we
decommissioned
our
out
cluster
this
year
and
we
record
logs
as
JSON.
We
send
those
two
data
dog
and
we
have
archives
stored
in
AWS,
s3
APM.
We
use
data
dog
for
network
performance,
we
use
data
dog
again
and
we
also
implement
thousand
eyes
that
gives
us
a
lot
of
insight
into
our
network.
Distributed
tracing
is
currently
a
work
in
progress
and
learning
a
lot
about
that
here
today,
a
lot
of
great
presentations
on
that
so
stats
t.
A
You
know
they
clearly
did
not
spend
a
lot
of
time,
making
a
shiny
logo
and
stess
t
itself.
You
know
it's
not
particularly
shiny,
but
you
know
it
makes
it
really
easy
to
measure
survey
and
service
performance.
It's
dead,
simple
and
it's
ubiquitous.
It's
it's
everywhere.
There's
tons
of
stats
to
clients.
We
also
use
stats
D
to
measure
client
performance
and
that's
a
little
bit
more
difficult,
but
and
for
a
couple
reasons
you
know-
take,
for
example,
our
chat
server.
We
want
to
measure
the
number
of
times
that
we
have
disconnects.
A
The
problem
is:
if
the
chat
server
disconnects,
you
can't
send
the
telemetry,
so
we
had
to
engineering,
storing
telemetry
in
the
client-side
storage
and
then,
when
the
chat
server
would
reconnect
we'd
send
that
up-to-date
dog
there's
also
caching.
How
do
you
deal
with
that?
And
then
you
have
a
large
set
of
browser
and
device
variants
when
you're
measuring
stats
T
data
at
the
client-side?
A
Well,
we
kind
of
follow
a
standard
approach
of
logs
traces
and
metrics.
You
know
the
the
three
pillars
of
observability
or
as
I
like
to
call
it
telemetry,
and
you
know
we
have
a
few
experts
there.
You
know
I'm
considered
metrics
experts.
You
know
we
have
lumberjack
Steve
who's
really
good
with
logs.
We
have
a
couple
folks
who
are
good
with
APM
and
you
know
so
our
expertise,
kind
of
breaks
down
by
team,
but
the
way
that
we've
tried
to
scale
it
out
to
a
thousand
engineers.
We
have
a
slack
channel
called
ask
Sree.
A
Where
folks
can
ask
us
questions.
We've
got
a
reliability
champions
group
which
meets
monthly
in
different
geographic
location,
urge
in
different
geographic
regions
where
we
talk
about
different
ability,
initiatives
and
we've.
We've
spent
a
lot
of
time
on
education.
You
know:
I
gave
an
observer
ability,
one-on-one
presentation
to
different
geographic
locations,
and
you
know
we
have
a
hands-on
with
data
dog
workshop.
A
A
A
A
Was
that
10
requests
or
was
that
10
million
requests
I,
don't
know
because
I've
already
aggregated
that
data
and
stored
it
and
what
you'll
find
is
several
vendors
out
there
have
bugs
in
their
percentile
tools,
and
you
know,
they'll
have
SLO
monitors
that
you
know
will
aggregate.
You
know
a
day's
worth
of
data
and
give
you
enough
time
for
that.
But
the
problem
is
you
can't
aggregate
that
over
a
month,
so
the
math
here
gets
very
tricky.
A
There
are
some
exceptions,
exceptions
to
aggregating
those,
but
it
goes
into
complicated
papers
with
the
math
is
really
over
my
head
frankly,
so
in
general,
we've
tried
to
avoid
using
percentiles
because
it
just
raises
too
many
questions.
So
what
do
we
do?
Instead?
We
use
counters.
We
count
numbers
of
requests,
we
count.
You
know
the
number
of
requests
under
our
s
Li
and
the
total
requests,
and
that
gives
us
all.
We
really
need
to
know
counters
are
easy
to
understand.
Yeah,
who
here
watch
Sesame
Street
grown
up
some
folks.
A
Okay,
all
the
older
folks
in
the
audience
counters
are
easy
to
implement.
You
know
stats.
T
has
you
know
increment
decrement
methods,
they're
easy
to
aggregate
when
you
store
counter
based
telemetry
in
a
time
series
database
that's
stored
in
one
timeslot,
you
can
take
that
and
another
set
of
time
slots
that
add
up
to
any
arbitrary
time
like
say
a
week
or
a
month,
and
you
can
sum
them
together
and
you'll
get
zero
loss
for
that
calculation
and
it's
easy
to
get
the
math
right.
You
know
you
add
them
up.
A
You
divide
them,
you
to
apply
very
simple
stuff.
A
lot
simpler
than
percent
tells
so
how
we
approach
this
for
SLS
we
use
counters
for
latency
SSL
is
say.
For
example,
you
know
my
SLI
as
my
request.
Time
is
less
than
500
milliseconds
I'll
count
up
all
those
requests
that
were
less
than
500
milliseconds
I'll
have
a
count
of
the
total
requests,
I'll
divide
them
and
I'll.
Add
my
success
objective
in
my
time
range
and
that's
it
I'll
get
an
answer
that
is
extremely
accurate.
A
Who
here
is
familiar
with
histograms?
Well
folks,
I'd
like
to
talk
about
these
a
lot
so
I
call
this
the
flexible,
latency,
SLI
approach.
We
capture
counts
of
requests
that
are
in
certain
bands.
That's
the
image
on
Wikipedia's
histogram
page.
You
can
see
two
different
implementations.
First,
one
you
know
call
it
the
you
know
ordinary
histogram
or
just
the
histogram,
where
we
would
count
up.
You
know
numbers
of
requests
that,
were
you
know,
between
100
and
200,
milliseconds,
200
or
300,
or
we
could
also.
A
We
have
an
implementation
that
we
can
use
a
cumulative
histogram,
which
is
essentially
what
Prometheus
used
where
you
have
a
running
count
of
the
requests
that
are
below
a
certain
value,
and
you
know
with
this
approach.
We
end
up
getting
one
time
series
for
each
of
these
latency
bands-
and
you
know
the
stat
C
metric
will
look
something
like
that
on.
The
bottom
will
have
a
name:
space
it'll
have
an
endpoint.
A
This
is
what
our
bands
look
like.
You
know.
We
have
every
10
milliseconds
from
10
to
100
milliseconds,
every
hundred
milliseconds
from
100
to
1;
second,
every
500
milliseconds
from
1
second
and
10
seconds,
and
every
five
seconds
from
10
seconds
to
60
seconds
and
for
an
example.
If,
if
I
record
a
latency
value
of
547
milliseconds,
my
metric
tag
will
be
la
600
in
the
cumulative
implementation
or
it
could
be
GT,
500,
less
than
or
equal
to
600
and
the
band
had
been
or
the
ordinary
histogram
implementation.
A
So
with
this
approach,
what
do
we
get?
We
get
relatively
low
errors
per
latency
band
say
here.
You
know
if
I'm
recording
something
between
one
second
and
1.5
seconds
and
1.5
seconds
to
two
seconds.
You
know
I
might
have
an
error,
you
know
a
plus
or
minus.
You
know
250
milliseconds
in
the
worst
case,
and
that's
not
bad.
You
know,
that's
that's
good
enough
for
what
we're
doing
with
a
ruby
based
web
service
to
get
really
good
results.
A
You
know
these
are
not
as
precise
as
HDR
histograms,
which
stands
for
high
dynamic
range.
There's
a
couple
notable
ones
out
there
guild
teens,
high
dynamic
range
histogram
and
the
cirque
onus
log,
linear
histogram,
both
of
those
use.
You
know
upwards
of
you
know:
40,
50
thousand
bins.
This
implementation
we've
only
got
40,
you
know,
so
it's
not
quite
as
precise,
but
it
gets
us.
You
know
80
90
percent
of
the
way
there
we
do
have
some
possible
cardinality
issues.
A
You
know
if
I've
got,
you
know,
say
a
thousand
or
eleven
hundred
in
points
in
my
app
and
then
I've
got
40
bins.
You
know,
that's
forty
thousand
possible
values
right
there.
You
know
multiply
that
you
know
times
the
number
of
hosts
multiply
that
times:
two
other
metrics
and
pretty
soon
you
know
you
might
have
a
couple
million
dedicated
time
series
for
this
approach,
but
you
know
if
it
takes
up
two
million
time
series.
That's
actually
not
bad.
Two
million
is
a
fairly
low
number
in
terms
of
time.
A
Servers,
databases
at
scale
and
the
nice
thing
about
this
approach
is
you
can
implement
it
with
any
monitoring
vendor
or
time
series
database
I
can
use
this
with
rrd
if
I
wanted
to.
Although
I
don't
know
why
I
would
so,
what
kind
of
data
can
we
get
out
of
this?
This
is
a
latency
band
graph,
where
each
line
is
representative
of
the
number
of
requests
that
were
within
a
certain
range.
A
So
you
know
here,
you
know:
I've
got
my
yellow
line
up
there.
That's
representing
you
know
the
number
of
requests
that
are
around
500
milliseconds.
You
know,
I've
got
my
purple
one
down
there.
That's
you
know
at
100,
milliseconds
and
just
looking
at
the
volume
I
can
find
that
99%
of
my
requests
fall
in
there.
So
what
can
I
do
with
that?
Well,
that
tells
me
I
probably
want
to
choose
my
SLI
initially
around
500
milliseconds,
you
might
say
like
well
can't
you
just
use
percentiles
to
do
that.
Again,
you
don't
know
the
volume.
A
Here's
another
approach
with
latency
bands
here,
we've
got
you
know
one
in
point.
You
know
we
clearly
have
a
request
volume
surge
in
there.
Is
that
something
for
concern.
Well,
looking
at
the
different
latency
bands,
I
can
see
that
you
know
my
light
blue
and
dark
blue
bands
at
the
top.
Those
are
the
lowest
latency
bands,
so
I
got
a
surge
of
a
lot
of
fast
requests,
so
the
system
performed
well
if
the
yellow
bars
in
the
bottom
had
expanded.
A
That
would
tell
me
my
service
slowed
down
a
lot,
so
this
is
a
great
visualization
that
I
can
use.
And
again,
you
know
this
is
something
that's
really
easy
to
implement
if
you
don't
have
access
to
high
definition
or
Hideout
high
dynamic
range
histograms.
This
is
a
good,
simple
approach,
with
simple
tools
that
you
can.
You
know
implement
for
a
lot
of
engineers
and
they
all
get
it.
A
So
you
know
we've
seen
you
know
how
we
can
define
these
error
budgets
we've
seen
how
we
can
implement
them
with
simple
tools,
but
what's
the
hard
thing,
so,
let's
go
take
a
look
at
you
know
brief
architecture
again,
you
know,
I
can
have
a
request
that
say
it
comes
into
my
CDN.
You
know
hops
through
my
proxy,
through
my
rails,
app
into
some
micro
services
and
for
each
of
those
I
can
get.
You
know
a
number
of
different
slis,
and
we
all
know
that
you
know
the
further
you
get
down
in
the
stack.
A
You
know
the
the
more
repeatable
you
want,
your
request
to
be,
but
you
know
when
I
initially
started
talking
to
folks
there
I
said
you
know
we
want
to
measure
what
the
users
see.
You
know
because,
as
I
mentioned
before,
you
know,
the
user
experience
is
very
different
than
what
you
might
get
at
a
certain
micro
service.
You
know
we
have
very
uses
patterns
from
different
users.
You
know
they
could
be
on
different
devices
on
on
different
networks,
and
so
really
we
want
this
sli
between
our
CDN
and
the
user
to
be
the
one.
A
That's
the
most
important,
because
that's
going
to
capture
everything
beneath
it,
you
mean
if
the
request
is
cached
it'll,
cache
that
if
the
request
goes
down
you
know
a
hundred
micro-services
it'll
represent
that.
So
really
the
key
thing
to
do
here
is
focus
on
the
users,
because
that's
what
matters
and
also
focus
on
the
audience's
that
you're
addressing
you
know,
I
can
have
my
success
objective
and
my
SLI
for
a
certain.
You
know
air
budget,
but
really
I've
got
five
different
era
budgets
that
I
didn't
need
to
construct
here.
A
You
know
my
a
saris,
a
my
Knock
need
to
have
a
time
range
of
five
minutes,
and
my
product
engineers
need
to
be
looking
at
that
for
an
hour.
My
product
managers
want
to
see
how
that's
doing
over
a
week,
VPS
want
to
know
about
a
month.
You
know
because
they'll
say:
hey
our,
you
know.
Trouble-Free
availability
for
this
month
was
99.99.
You
know,
and
my
you
know,
c-suite
will
be
looking
at
these
numbers
over
a
quarter.
A
So,
let's
sum
it
up,
you
know
how?
How
did
we
implement
our
budget
successfully
over
a
thousand
engineers
and
about
four
months
give
everyone
a
formula
make
it
simple
use,
simple
tools
that
can
give
you
rich
data,
use
latency
bands
in
particular
histograms
to
grab
duration
data
measure,
your
slis
to
the
client
as
close
as
possible
and
use
our
budgets
with
appropriate
time
ranges
for
different
audiences.
A
A
A
So
the
question
was:
if
we
have
a
set
of
chain
services,
do
we
record
just
the
deepest
service
or
all
of
them?
And
my
answer?
Is
we
record
all
of
them
as
much
as
possible?
We
tried
to
record
in
some
cases
you
know
redundant
telemetry,
you
know,
will
collect
service
times
using
metrics
and
also
APM
and
we'll
compare
them.
You
know.
That's
actually
yielded
points
where
we
found.
A
You
know
our
APM
wasn't
running
in
one
point
of
delivery
or
you
know
some
of
our
metrics
were
off
there,
and
this
is
you
know
this
is
something
I
really
had
to
push
on,
because
you
know
folks
said
like
hey,
you
know,
you're
wasting
data
and
I
said
well.
You
know
there's
a
reason
that
the
Space
Shuttle
had
five
different.
You
know
redundant
systems
now
because
when
something
fails
you
know,
we've
got
a
backup.
C
Hey
Fred
thanks
a
lot.
If
you
follow
the
Google
SRA
model,
one
of
the
things
that
they
use
Arabic
to
drive
is
very
specifically
deciding
if
a
product
team
is
going
to
work
on
features
or
work
on
reliability
and
sort
of
views
as
an
absolute
decision
point
I
think
that
can
be
harder
in
smaller
organizations.
What
does
n
des
decided
to
do?
Do
you
drive
those
decisions
from
the
air
budget
and
do
you
stick
to
that
rigorously
or
do
you
make
that
a
case-by-case
decision
so.
A
The
question
is,
you
can
hear
that,
unlike
so
we
as
part
of
this
effort,
we
not
only
implemented
the
technical
phases,
but
we
had
each
team
develop
an
out
of
error
budget
process,
so
it
was
basically
a
document
saying
like
hey.
If
our
error
budget,
you
know,
is
exhausted,
here's
how
we're
going
to
prioritize
work,
so
that
was
part
of
the
four
effort
and
it's
actually
one
of
the
most
important
parts
of
it.
Even
though
it's
not
the
most
technically
involved.
B
Hi
so
with
this
implemented
like
a
Zendesk,
a
team
by
team
basis
like
they
started
with
a
few
teams
first,
and
was
it
like
a
top
down
from
the
top
or
was
a
grassroots
started
from
engineering?
Then
management
took
it
out
like
how.
How
were
you
able
to
implement
it
across
the
whole
organization
in
the
timeframe?
So.
A
It
was
an
initiative
that
came
down
from
the
top.
You
know,
because
part
of
you
know
part
of
our
mission
is
reliability
is
a
core
feature
of
ours,
so
that
was
supported
all
the
way
you
know
down
from
the
CEO
to
the
SVP
of
engineering
all
the
way
down
to
the
team's.
How
did
we
sorry?
What
was
the
second
part
of
the
question?
Oh,
how
did
we
roll
it
out?