►
From YouTube: Better Reliability Through Observability and Experimentation - Julie Gunderson & Kerim Satirli
Description
Don’t miss out! Join us at our upcoming hybrid event: KubeCon + CloudNativeCon North America 2022 from October 24-28 in Detroit (and online!). Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Better Reliability Through Observability and Experimentation - Julie Gunderson, Gremlin & Kerim Satirli, HashiCorp
Site Reliability Engineering (SRE) treats reliability as a software problem, but it really is an organizational problem that requires a different mindset. When the reliability of our service drops, so does our ability to create value for the organization we represent. In this talk, Julie and Kerim will take the audience on a guided journey, starting with how to determine if and how workloads are misbehaving and ending with practical approaches to improve reliability. Through simulated outages (of all types!), observability, and analysis, Julie and Kerim will show attendees how to catch and prepare for service disruptions. Going beyond deployments, attendees will also learn how to combine OpenTelemetry and OpenTracing to instill reliability into their systems.
A
I
think
we're
just
going
to
go
ahead
and
get
started.
It
does
look
like
there's
a
line
outside
just
to
let
you
all
know
who
I
am.
I
am
julie,
gunderson
a
senior
reliability
advocate
at
gremlin,
I'm
actually
an
organizer
of
devops
days,
boise,
I'm
an
avid
mushroom
hunter,
but
those
are
morel,
mushrooms
and
I
arrived
in
spain
by
plane.
B
And
my
name
is
kareem
satiri,
I'm
a
senior
developer
advocate
at
hashicorp,
I'm
a
huge
fan
of
aerial
photography
or
one
of
those
guys
with
drones,
very
annoying
used
to
be
a
conference.
Organizer
gave
up
on
that.
Thank
you.
Kovit
also
arrived
here
by
plane.
So
I
know
we
all
had
lunch,
but
quick
show
of
hands.
Movement
is
very
important
to
help
with
digestion
who
came
here
by
plane.
Raise
your
hand,
oh
geez,
okay,
oh
wow,
that
83
percent
something.
B
Yeah
we'll
go
80,
83
percent
doesn't
truly
matter
because
you
know
metrics
are
always
a
lie,
but
flying
obviously
one
of
the
safest
ways
of
traveling,
and
yet
you
still
might
encounter
an
incident
or
two
sorry
an
unscheduled
event.
My
airline
does
not
want
to
call
it
an
incident
in
itups.
We
refer
to
it
as
an
outage
as
an
incident,
but
to
keep
everyone's
stress
level
down,
especially
after
seeing
so
many
hands
up
we're
going
to
avoid
harmful
language
like
that,
we.
B
So
picture
this
and
it's
purely
hypothetical,
this
never
happens.
Your
plane
is
on
a
landing
approach
like
the
plane
behind
us.
This
is
the
amsterdam
tower.
We
try
to
find
a
sky
that
reflects
valencia's
at
night,
it's
very
hard
because
amsterdam
is
usually
very
grey
crew
in
that
flight.
Deck
just
reconfirmed
the
correct
one
way.
B
B
B
A
And
they
don't
panic
because
they
are
actually
practicing
for
these
scenarios
all
the
time
now
I
will
take
a
step
back
and
when
you
do
see
your
flight
crew
all
of
a
sudden
start
to
buckle
up.
Oh
maybe
that's
when
you
should
pay
attention
a
little,
but
during
this
they
know
what
to
expect.
During
these
situations,.
B
B
A
So
one
of
you
answered
that
observability
is
akin
to
a
surveillance
state
and
we're
just
gonna
kind
of
have
to
come
back
to
that
one
in
a
little
bit
now
more
than
one
person
with
maybe
the
title
of
sales
or
account
executive
suggested
that
observability
is
just
an
add-on
sku
that
they
sell
and
it
gives
you
that
automated
visibility,
so
all
you
have
to
do,
is
install
observability.exe
double-click
and
everything
is
instrumented.
Super
easy.
B
B
B
B
A
B
A
B
A
And
so
you
want
to
think
about
this
process
as
a
stand-in
for
any
engineering
product,
we
start
building
a
thing
and
when
we
start
to
build
that
thing,
the
first
thing
that
we
need
to
do
is
we.
We
gather
the
requirements
right,
so
we
want
to
understand
what
are
our
customer
needs
we
meet
with
those
stakeholders,
we
document
the
outcomes.
A
We
actually
go
through
quite
the
process
for
this,
but
just
don't
have
an
entire
day
to
talk
about
it,
but
once
we've
done
that
research
and
once
we've
established
those
timelines,
then
we
start
building
that
thing.
So,
in
the
case
of
that
plane
we
know
how
many
passengers
it
needs
to
hold.
We
know
how
far
that
plane
needs
to
go
what
the
fuel
requirements
are
and
when
we
look
at
that
those
are
the
things
that
we
are
aware
of
that
we
know
now.
B
B
A
B
A
Or
else
indeed,
so
we
want
to
think
about
that
and
that's
what
we're
talking
about
today
with
observability.
Are
we
able
to
understand?
Can
we
look
from
the
outside
in
and
be
able
to
understand
that
we
are
experiencing
an
incident,
so
the
or
else
that
kareem
was
mentioning
is
two
in
the
united
states,
two
air
force
trained,
observability
specialists,
otherwise
known
as
fighter
jet
pilots.
A
So
if
that
context
is,
is
lost
for
long
enough,
two
pilots
will
physically
fly
up
and
flank
the
plane
to
observe
from
the
outside,
in
that
it's
still
there.
So
not
an
ideal
situation.
Also,
if
you
are
on
a
plane-
and
you
do
see
two
fighter
jets-
don't
worry
it's
actually
part
of
the
process.
So
it's
okay,
but
once
you've
detected
the
incident,
that's
now
when
you
can
actually
start
working
to
restore
that
service,
and
that's
excellent,
so
we're
going
to
walk
through
this
here
on
how
we
get
there.
B
B
It
does
actually
cause
problems,
sometimes
after
flying
so
remember
we
owed
you
one
more
answer,
and
this
was
the
one
that
we
got
from
most
people.
Observability
is
really
the
information
you
didn't
think
you
needed,
but
that
could
actually
solve
your
problem
and
these
are
the
unknown
unknowns.
It's
easy
to
build
for
the
stuff
for
the
happy
path,
even
for
the
imagined
unhappy
path,
but
there's
always
the
stuff
that
you
can't
truly
anticipate
until
you've
experienced
it
and
it
is
no
longer
unknown.
B
Exactly
code
is
poetry,
and
your
error
messages
should
be
haikus
that
help
you
understand.
What's
going
on,
put
the
things
in
there
that
make
sense
for
you
and
keep
them
consistent
if
your
system
is
complex
enough,
which
obviously
it
is
put
in
numbers
that
you
can
track
in
terms
of
error
codes,
a
400
will
not
always
help
you
detect
that.
Yes,
the
client
didn't
provide
great
data,
but
it
doesn't
necessarily
mean
that
nothing
they
provided
was
good.
A
Now
that
said,
we
also
have
tracing
and
tracing
helps
to
helps
you
deal
with
the
complexity
of
your
distributed
systems.
So
look:
there's
lots
of
tracing
tools
out
there
and
you're
you're
tracing
tool
of
choice.
That's
going
to
help
you
understand,
maybe,
where
that
single
user
request
goes
and
subsequent
requests.
It
creates
throughout
your
system,
so
following
a
single
user
through
a
distributed
system
with
multiple
dependencies,
it's
incredibly
complex,
but
that
actually
gives
you
the
insight
into
how
your
application
is
behaving
within
that
web
of
both
upstream
and
downstream
dependencies.
A
How
did
you
know
if
you
should
wake
up
patrick
at
two
o'clock
in
the
morning
on
a
tuesday
or
at
three
o'clock
in
the
morning
on
a
saturday
after
he'd
been
hanging
out
at
cubecom
for
the
whole
week
or
if
it
was
something
that
could
wait
until
a
normal
work
day?
So
when
we
talk
about
measurements,
we
need
to
understand
how
our
system
is
working,
how
severe
those
incidents
are,
but
also.
B
A
A
So
let's
go
ahead
and
go
back
and
look
at
that
timeline
of
the
incident.
So
if
we
look
at
the
orange
line
here
that
represents
this
extended
period
of
time
where
we
didn't
even
know,
we
were
in
that
time
to
detect
the
incident
so
ttd
right
and
that
could
have
resulted
from
many
things.
Maybe
our
monitoring
solutions,
maybe
they
weren't
working,
maybe
they
weren't
picking
that
up.
A
What
we
need
to
know
is
where
our
systems
are
having
those
interruptions,
and
that
comes
back
to
observability,
and
so
with
that
time,
to
detect
being
so
long,
we
want
to
shorten
that
and
we
want
to
shorten
that
with
observability
with
monitoring
and
then
with
experimentation,
because
we
can't
fix
any.
We
can't
restore
any
of
our
services
if
we
don't
know
that
they're
broken
first.
So
when
we
talk
about
experimentation,
we're
really
talking
about
science
here,
because
science
and
chaos,
chaos
might
be
a
little.
A
Unfortunately,
named
people
tend
to
get
a
little
bit
freaked
out,
especially
c
levels.
They
don't
love
hearing
chaos,
they
don't
love
hearing
failure.
But
when
we
talk
about
chaos,
engineering,
we
do
follow
the
scientific
method.
So
we
start
with
you,
know,
observation
and
baseline
metrics,
because
it's
important
to
understand
how
your
systems
behave.
Now,
some
of
you
might
call
that
a
steady
state,
I
prefer
nominal
state
because
your
systems
are
going
to
vary
depending
on
on
what
you're
experiencing.
A
After
we
form
that
hypothesis
we're
going
to
design
an
experiment
around
that
and
when
we
design
those
experiments,
it's
really
important
to
understand
the
blast
radius.
So
the
blast
radius,
if
you
think
about
it,
is
how
big
that
experiment
is
going
to
be
how
much
of
your
system
is
going
to
be
impacted.
So,
for
example,
your
weekend,
oh
yeah,.
B
A
One
of
the
other
things
you
want
to
have
in
place
is
abort
conditions,
so
this
is
when
you're
going
to
say.
Oh,
we
just
learned
something
about
our
system
that
we
didn't
expect
and
we
better
stop.
This
experiment
netflix
famously
uses
stream,
starts
per
second
as
a
metric
and
they're
using
a
metric.
That
is
a
direct
reflection
of
their
customers
when
they're
running
experiments
and
they
see
that
stream
starts
per
second
start
to
drop.
A
That's
when
they
halt
that
experiment,
because
while
we
are
all
having
fun
playing
around
with
our
systems,
we
have
to
remember
that
there
is
somebody
on
the
other
end.
Who
is
ideally
trying
to
use
that.
So
we
want
to
keep
the
user
in
mind.
This
is
all
about
making
their
experience
better
and
now
that
you've
run
your
experiment,
you
can
just
walk
away
you're
done,
but
not
because
you
want
to
take
a
look
at
those
results
and
there's
going
to
be
likely
one
of
two
outcomes.
A
So
either
you
validated
your
hypothesis
yeah,
your
systems
are
reliable
or
they
they
failed
gracefully
or
you
found
something
in
your
system.
You
found
a
potential
failure
point.
Luckily,
before
your
customer
did
so
analyze
those
results,
because
then
you
can
go
back
and
either
have
confidence
that
your
system
works.
The
way
you
expected
it
to
or
now
you
know
what
you
need
to
do
to
harden
your
system.
A
A
So,
first
of
all
what
we're
learning
on
our
teams.
We
should
share
with
other
teams,
so
they
can
also
learn
how
to
harden
their
systems,
but
we're
also
learning
as
an
organization
and
developing
that
culture,
where
we
embrace
failure,
we're
developing
that
culture
where
we
want
to
learn
from
each
other
and
the
way
we
do
that
is
we
share.
We
share
wins.
We
share
our
failures.
We
share
our
learnings.
B
B
Somebody
had
taught
you
at
that
point
help
build
that
culture,
no
matter
how
big
your
organization
is
small
contributions
like
that
go
a
long
way,
and
so,
as
we
learn
from
failure,
we
want
to
generally
look
at
four
signals
in
the
sre
book
and
the
site
reliability
engineering
book
from
google.
This
would
be
called
the
four
golden
signals
if
you
like
a
different
color
totally
fine
before
we
continue
a
word
of
warning
learning
together
is
awesome.
Learning
because
you
caused
a
huge
outage,
less
awesome.
B
B
So
let's
talk
about
latency
a
little
bit
the
time
it
takes
to
serve
as
a
request.
So,
for
example,
with
our
plane
are
the
voice
comms
between
plane
and
tower
just
delayed
or
is
something
else
going
on.
There
are
a
couple
of
ways
you
can
simulate
this
injecting
delays.
Programmatically
super
easy.
If
you're
an
early
career
engineer,
that's
comes,
you
know
it's
part
of
the
job.
B
I've
I've
definitely
been
there.
You
could
also
build
in
some
sleep
commands
or
whatever
your
language
of
choice,
offers
you
could
change
your
dns
in
network
settings,
thoughts
with
the
timeouts
black
hole.
Certain
connections
makes
routing
really
really
complex
if
you're
codifying
your
infrastructure,
this
is
very
easy
to
establish
and
make
go
away
again.
It's
a
good
thing
and
then
super
fast
way
to
introduce
latency,
no,
no
pun
intended.
B
B
B
A
Actually,
I
just
want
to
pause
here
so
this
time
thing.
That
is
an
error
that
we
would
think.
Okay,
our
systems
can
handle
that.
I
just
have
to
let
you
know
every
time
we
have
daylight
savings
time
in
the
states.
My
card
for
my
bank
doesn't
work
now.
Luckily,
I'm
too
lazy
to
have
to
like
change
all
my
payment
methods
on
netflix
and
hulu
to
go
to
another
bank,
but
we
have
to
remember
that
our
customers
can
make
changes
like
that.
B
Actually
glad
the
european
banking
system
is
a
little
more
stable,
so
yeah.
When
we
talk
about
traffic,
we
refer
to
the
demand,
that's
being
put
on
a
system.
This
isn't
exclusively
http
traffic,
anything
that
goes
over
wire.
Maybe
you
have
an
internet
of
things
setup.
Maybe
all
your
stuff
goes
through
the
ether.
B
All
of
this
is
traffic,
and
eventually
that
can
catch
up
with
your
system,
create
some
spikes
for
http
traffic.
Super
easy
use,
something
like
a
b
or
newman.
If
you
build
apis,
just
spawn
tons
of
instances
and
see
how
your
apis
respond
to
that
see
how
your
web
application
firewall
responds
to
that,
if
it
actually
blocks
your
own
requests
and
then
disable
those
blocks,
because
your
attackers
will
do
the
same
thing,
they
will
figure
out
ways
around
that
and
will
not
be
caught
by
the
first
level.
B
This
will
give
you
enough
traffic
to
test
with.
Hopefully,
if
not,
you
might
want
to
ask
why
you
have
three
then
for
saturation,
we're
looking
at
system
utilization
and
what
constraints
our
system
is
experiencing,
and
this
is
a
tricky
one,
because
this
is
very
very
specific
to
your
system.
We
had
somebody
come
up
to
us
yesterday.
Ask
about
what
a
good
cpu
is
for
an
architecture.
They
were
planning
and
it's
it's
a
guess
right.
B
You
could
start
out
by
altering
the
scaling
logic
of
your
orchestrators
delay.
Scale
outs
to
create
hot
zones,
not
the
covit
kind
and
force
more
load
onto
a
single
system,
or
you
could
fill
up
disks
with
randomly
generated
data
if
you
ever
played
with
linux,
you've
probably
found
ways
to
do
that
and,
of
course,
there's
plenty
of
different
approaches.
B
I
think
the
person
that
suggested
what
was
it
observability.exe?
You
could
use
consume.exe
to
just
completely
burn
through
your
windows:
cpu,
sorry,
your
cpu
on
a
windows
system.
Obviously
it's
the
same
cpu
and
so
latency
errors,
traffic
saturation,
the
four
primary
signals
you're
looking
for
and
while
we
present
them
as
individual
things,
they're
very
much
connected.
B
If
you
shift
your
traffic
from
a
single
from
two
instances
to
a
single
instance,
you're
going
to
get
more
disk
usage,
you're
going
to
get
more
saturation,
your
disk
is
going
to
get
fuller
and
eventually
you're
going
to
get
errors,
because
your
application
is
no
longer
able
to
write
and
we're
talking
about
this
in
a
instant
setup.
But
all
of
this
holds
true
for
pretty
much
any
setup
that
you
can
imagine.
A
A
big
part
of
this
is
understanding
our
systems
better,
but
also
building
that
culture.
So
one
thing
that
we
talk
about
is
game
days
and
with
game
days.
You're
practicing
chaos,
engineering
experiments,
but
not
just
with
your
team
with
other
teams
in
the
organization.
It's
always
a
good
idea
to
at
least
include
one
other
team,
because
you're
going
to
learn
about
how
their
systems
work
they're,
going
to
learn
about
how
your
systems
work
and
you're
also
going
to
learn
how
to
work
together
and
have
that
cross-functional
communication.
B
So
deep
breath
we're
nearly
at
the
end,
so
I
want
to
give
you
some
takeaways.
First
of
all,
everything
we
talked
about
today
is
tool
agnostic.
It
doesn't
matter
if
you're
in
a
container
context
or
not,
it
doesn't
matter
if
you're
on
physical
hardware
or
virtualized
hardware,
which
is
just
more
expensive
physical
hardware,
complex
systems
are
ultimately
built
by
humans
and
those
humans
are
the
primary
contributor
to
lowering
your
error
budget,
but
those
same
humans
also
can
be
also
the
primary
contributor
to
increasing
your
error
budget
and
allowing
you
more
time
to
experiment.
B
We
gave
you
this
heads
up
before,
but
everything
you
build
should
be
codified.
It
is
so
much
easier
to
look
at
code
in
the
broadest
sense
of
the
word.
A
protocol
is
also
codification
to
look
at
that
and
understand
how
to
restore
your
system
when
you
codify
you're
removing
click
ops,
that's
a
good
thing.
B
A
B
And
finally,
none
of
this
is
really
a
technical
problem.
This
is
very
much
a
people
problem,
so
remember
when
we
said
teach
that
new
engineer
teach
that
new
person
on
your
team,
how
things
work
create
that
culture
of
reliability
create
that
culture
of
it's
okay,
to
fail,
it's
okay
to
ask
and
it's
okay,
to
learn
together.
A
B
B
B
And
then
I
think
we
were
asked
to
put
this
in
there.
So
please
enjoy
definitely
take
pictures
of
this.
I
see
you
pulling
your
phones
down.
That's
not
okay,
keep
them
up.
You're
gonna
need
them
for
the
next
slide,
though,
hit
us
up
at
our
booths.
We
have
socks
at
hashicorp
super
nice
when
it's
not
this
balmy,
and
if
you
want
a
copy
of
these
slides,
we'll
upload
this
as
soon
as
we
get
wi-fi
back
and.