►
From YouTube: From Monitoring to Observability: Left Shift your SLOs with Chaos - Michael Friedrich, GitLab
Description
Don’t miss out! Join us at our upcoming hybrid event: KubeCon + CloudNativeCon North America 2022 from October 24-28 in Detroit (and online!). Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
From Monitoring to Observability: Left Shift your SLOs with Chaos - Michael Friedrich, GitLab
Security has shifted left in CI/CD pipelines. Traditional service monitoring moved on with metrics, logs and traces and observability embraces the unknown unknowns. Developers and SREs are instrumenting applications with distributed tracing. How do service level objectives (SLOs) add to the bigger picture? This talk invites into a developer’s tale about ops deployment scalability, availability threshold definitions and measuring application performance. What are the benefits of app instrumentation, metrics and traces and where does the journey start? Dev becomes Ops: SLOs need to be well understood and simulated early in the development process. New building blocks come to play: Continuous Delivery, quality gates and chaos engineering - is it possible to left shift SLOs with Chaos in your CI/CD pipelines?
A
Hi
everyone
nice
to
finally
finally
meet
you
in
person.
Now
I'm
a
little
nervous,
but
it's
really
I'm
really
happy
to
be
here
and
to
talk
about
a
little
bit
about
from
monitoring
to
observability
left
shift
your
slos
with
chaos.
Today,
I'm
michael
I'm
a
senior
developer
evangelist
at
gitlab
on
the
internet.
You
can
find
me
as
dns
mihi,
which
is
dns
michi,
it's
the
lovely
form
of
michael
in
german,
and
I
figured
later
on.
A
A
while
ago
or
like
10
years
ago,
we
had
certain
things
like
black
box
monitoring
state
changes,
measuring
slo
sla
reporting
over
time.
At
a
certain
point,
metrics
have
been
added,
trending
service
level
objective
have
been
defined
and
we
moved
into
certain
things
like
white
box
monitoring
and
one
of
the
things
which
inspired
me
in
my
past
life
as
an
open
source
monitoring.
Maintainer
myself,
the
promisius
metrics
endpoint
was
added
to
docker
and
it
made
it
much
more
easy
and
convenient
to
look
inside
the
application
from
there.
A
A
At
a
certain
point,
you
needed
to
instrument
the
code
like
making
a
code
change
to
really
export
the
metrics.
You
wanted
to
see
the
term
or
site
reliability,
engineer,
sre
was
invented,
and
maybe
it
solved
everything
maybe
not,
but
in
the
end
it
was
really
a
nice
way
to
move
forward
and
go
into
the
metrics
direction
as
a
developer.
On
the
other
side,
something
goes
wrong.
A
There
was
memory,
corruption.
Sometimes
it
was
exhausted,
so
maybe
a
leak
or
something
like
that
and
it
behaved
differently
on
operating
systems
like
on
windows.
There
were
stack
guards
which
caused
the
crash
and
when
there
was
a
security
scanner
running,
it
was
a
different
crash
and
this
was
super
hard
to
debug
and,
to
be
honest,
it
burned
me
out
in
2019
and
I
thought
well
oops.
A
A
Maybe
the
api
clients
and
the
connections
could
have
used
something
like
chaos,
engineering
or
fussing,
or
something
else
to
really
like
figure
out
if
the
problem
is
being
hit
now.
Another
thing
is
like
thinking
about
switching
rows
into
ops
and
my
nickname
is
dns
mickey
and
I'm
saying
it's
always
dns.
It's
probably
true.
A
I
was
working
at
the
university
of
vienna
back
then,
and
there
was
a
rollout
with
the
dot
id
domain
with
dns
sec.
We
had
the
signing
hardware,
we
had
a
state
machine
of
steps.
On
a
friday
afternoon,
a
script
was
changed.
It
was
deployed
to
production,
the
signing
stopped,
no
dns
updates.
Whenever
you
registered
a
dot
id
domain,
nothing
happened.
A
Monitoring
was
in
place,
which
meant
like
we
did
monitor
things
we
had
the
serial
and
the
offset
defined.
The
first
alarm
came,
I
think,
on
saturday
at
3
a.m
in
the
morning
via
email.
So
I
wasn't
reading
that,
but
at
4
am
there
was
a
flood
of
sms
messages,
text
messages
from
all
the
name
servers
involved.
I
think
they
were
25
at
the
time.
Every
minute,
25
sms
is
not
fun
and
like
after
the
fifth
or
whatever
alarm,
it
wasn't
really
fun
like
logging
into
terminal
and
trying
to
figure
out
what
is
going
wrong.
A
A
So
the
idea
was
to
say
maybe,
as
a
retrospective,
now
thinking
of
having
a
staging
environment
having
everything
rolled
with
github
so
infrastructure
as
code
persisted
in
git
and
verified
this,
that
the
changes
are
being
tested
and
that
that
no
other
things
are
going
wrong.
The
sli,
for
example,
could
be
the
zone
serial
h.
The
slo
is
a
certain
way
of
saying:
hey
the
zone
should
be
not
much
older
than
one
hour
and
I
thought
of
well
chaos.
A
Another
story:
switching
gears
from
dev
to
orbs
to
devops,
like
we
are
using
we're
used
to
using
containers
and
one
of
the
things
which
happened
in
I
think
it
was
in
september
2020
when
docker
announced
hub
for
docker
hub
the
rate
limits
and
we
didn't
know
what
is
happening.
Is
my
ci
cd
pipeline,
not
working,
because
I
cannot
pull
the
containers.
A
What
about
cloud
native
deployments
rolling
out
your
application
in
kubernetes,
the
the
left
hand,
side
or
the
first
container
works,
and
the
second
doesn't
what
is
what
is
going
on
and
similar
to
like
when
it's
based
on
an
ip
address?
What
what
happens
to
organizations
using
an
art
or
bigger
cloud
providers
and
whatnot?
A
A
This
was
solvable,
the
unknown
state
was
like.
Well,
maybe
there's
something
in
the
logs.
Maybe
there
is
something
else
which
tells
me
too
many
requests,
but
I
need
to
like
dig
deep
down
into
a
ci
cd
pipeline
interface
or
specific
anywhere
else,
and
the
problem
which
could
have
occurred
is
like
the
application
or
the
shop
is
presenting
a
specific
price,
but
to
only
a
third
of
your
customers
and
they
say:
well,
it's
awesome
it's
on
sale,
but
actually
it.
Oh
sorry,
though,
those
who
are
seeing
the
new
price
they
think.
A
Now.
Thinking
of
this
one
could
say:
well,
we
are
defining
the
limit
as
a
service
level
indicator
and
the
objective
should
be.
The
pull
counts.
Remaining
should
be
ten
or
maybe
like
in
a
bigger
environment,
a
hundred
or
a
thousand,
or
something
like
that,
and
when
the
slo
is
failing,
we
shouldn't
even
start
a
deployment,
because
when
we're
looking
at
ci
cd
deployments
as
a
developer
and
saying
yeah,
nothing
happens,
and
I
don't
know
what
to
do
this.
A
This
shouldn't
be
needed
and
it's
it
can
be
quite
frustrating
if,
if
nothing
works
now,
the
idea
is
to
define
service
level
objectives
and
the
thing
is
like
okay.
What
is
that
and
where?
Where
do
I
start
like
as
an
s3,
devops
devops
and
one
like
it's
we're
going
into
monitoring?
We
can
go
into
metrics,
we're
defining
keys
and
tags.
A
We
have
values,
but
what's
next
and
one
of
the
things
is
like
to
get
started
more
easily,
I
found
promises
and
prom
coil
are
pretty
good,
which
allows
you
to
really
like
collect
the
metrics.
You
can
query
the
metrics.
You
can
combine
certain
logical
operators
with
them
and
it's
it's
an.
I
would
say
a
rather
easy
language
to
learn
and
it's
a
way
to
define
your
several
service
level
objectives
and
verify
they
are
not
violated.
A
When
you
have
a
customer
application
or
your
application,
you
might
be
needed
to
instrument
the
code
now
defining
the
slo
with
promcal
can
be
done
in
a
way
of,
for
example,
using
alert
rules
which
are
also
then
triggering
alerts,
and
it
also
allows
you
to
define
the
errors
basically
which
are
allowed
in
the
error
function.
So
when
there's
a
specific
point
of
like
I'm
pinging
a
service
or
I'm
using
a
probe
exporter
to
verify
that
something
is
reachable
over
time,
I
really
want
to
ensure
that
it's
99.9
and
when
the
slo
is
violated.
A
How
can
I
benefit
from
service
level
objectives
similar
to,
for
example,
shifting
security
left
and,
like
one
of
the
ideas,
was
okay,
we
are
we're
using
slos,
we
do
have
chromius,
we
are
calculating
the
alerts.
We
have
the
prom
cals,
which
can
also
be
written
in
the
openslo
format.
So
this
is
a
new
specification
which
was
defined.
I
think
last
started
last
year
in
may
and
now
reached
version.0
last
week
or
something
and
using
that
knowledge
we
can
deploy
it
into
ci
cd
or
use
using
environments.
A
Captain
itself
works
as
a
quality
gate,
but
also
like
as
a
it
is
more
than
a
quality
gate.
Actually
you
can
do
more
with
like
ensuring
a
specific
state.
Is
there
it's?
I
would
say
it's
basically
an
observability
platform
for
continuous
delivery
as
well.
It
has
a
ui
where
you
can
test
certain
things.
Certain
workflows-
and
I
would
really
encourage
you
to
try
it
out
and
see
whether
a
quality
gate
in
your
cicd
pipeline
makes
sense
to
to
measure
the
slos.
A
The
thing
which
came
around
was
like
we,
I
have
the
quality
get
now
and
we
know
how
to
use
slos
with
prometheus,
but
the
problem
is
like:
how
could
I
like
simulate
a
production
incident
so
like
fail,
the
database
fail,
failure,
connection
fail,
dns
see
what
is
going
on,
and
so
I
can.
I
learned
about
chaos.
A
Engineering
like
not
adding
it
only
to
the
production
environment,
but
also
deploying
a
staging
environment
or
test
environment
from
a
much
request,
letting
it
run
for
for
some
minutes,
generate
some
chaos
and
we'll
come
to
that
in
a
bit
and
then
see
how
the
application
is
behaving.
Is
it
crashing?
Is
there
some
certain
like
deadlock
or
something
else,
and
if
I
really
want
to
dive
into
monitoring
and
collecting
metrics,
I
can
also
trigger
alerts.
I
can
integrate
it
into
much
requests.
I
can
use
alerts
and
internet
managements,
and
so
on.
A
A
The
thing
is
I
started
with
like
cloud
network,
and
then
we
have
a
cluster
with
kubernetes,
where
we
can
deploy
our
workloads
and
deployments
and
so
on.
We
can
use
a
chaos
framework
which
defines
experiments
and
it
might
be
having
instrumentation
sdks,
where
I
can
write
my
own
chaos,
my
own
rules,
what
should
be
happening-
and
I
found
these
like
terms
to
keep
in
mind-
really
helpful-
to
really
keep
going
and
diving
deep
intruded
into
the
tools
into
the
frameworks
which
are
also
cncf
projects.
A
One
of
them
is
litmus
chaos
which
I
think
I
found
last
year,
and
we
also
had
the
folks
in
our
everyone
can
contribute
cafe
meetup
a
while
ago.
The
idea
is
to
fail
your
infrastructure
and
cluster.
You
want
to
see
how
the
application
is
behaving
like
I
just
said,
and
see
and
verify
the
server's
service
level
objective.
A
If
it's
really
matching
and
from
there
defining
the
actions
and
improvements,
it
provides
a
nice
ui,
the
getting
started
guide
is
awesome,
so
it's
like
really
like
five
minutes
and
then
you
can
get
started
and
there
is
like
the
chaos
hub
with
experiments
workflows.
The
communities
is
amazing,
like
helping
each
other
and
adding
more
things.
One
doesn't
need
to
invent
by
yourself,
which
is
a
good
thing
in
in
like
chaos,
engineering,
and
I
think
it's
like
I
really
like
the
ui.
A
A
A
Ip
addresses,
or
it
just
provides
ipv6
and
then
see
how
it
behaves
yeah
and
for
like
the
devops
story,
maybe
having
a
register
or
container
registry
proxy,
which
does
some
sort
of
limiting
and
rate
limiting
in
there
and
just
seeing
how
this
actually
behaves
and
then
defining
the
actions
you
want
to
do
when
you
want
to
dive
deeper
into
this.
Like
your
own
chaos,
there
are
experiments
sdks
out
there
and
I
think
litmus
has
go
and
python.
If
I'm
not
mistaken,
you
can
integrate
it
into
ci
cd.
A
A
The
thing
is,
there
are
some
limits
with
chaos
or
keras
engineering,
because
it
consumes
more
resources,
it
might
harm
different
teams
and
different
workflows.
So
it
might
have
some
impact
on
the
system
as
well,
so
don't
try
to
enable
chaos
everywhere
and
then
see
everything
break.
I
think
this
kind
of
needs
some
planning
and
some
maintenance
and
also
awareness
that
this
is
now
being
deployed
in
in
the
systems,
and
the
other
thing
is
to
keep
in
mind.
A
Now
the
thing
is,
I
thought
about
my
own
story
with
the
failing
connections
and
dns
failing
and
things
like
that
and
the
memory
leak,
and
so
I
hacked
a
quick
dimmer
for
which
uses
a
c
plus
plus
code.
It
gets
deployed
to
kubernetes
q
promises
does
all
the
monitoring
and
chaos
mesh,
invokes
some
chaos
and
there's
potential
to
use,
slows
and
alerts,
and
so
on
and
the
image
on
the
right
hand
side.
I
generated
that
with
a
small
tool
yeah.
A
So
in
this
case,
what
I'm
trying
now
as
a
short
demo,
I
have
created
or
written
a
short
application.
Let
me
see
the
application
is
actually
it's
a
short
c
plus
plus
code.
You
don't
need
to
understand
everything
which
is
going
on
in
here.
The
idea
is
to
create
a
buffer.
Do
some
dns
resolving
and
then
error
out
when
it's
successful
it
does
something
and
then
it
deletes
the
buffer.
A
So
at
this
specific
point
there
is
a
condition
missing
to
delete
the
memory
all
the
time,
so
I
may,
at
the
the
error,
is
intentionally
of
course,
for
this
for
this
demo
now.
But
it
kind
of
reminded
me
where
I
was
at
the
thing
I've
prepared
already
is
this:
application
is
built
in
gitlab
ci
cd
as
a
docker
image,
and
the
manifests
have
been
deployed
into
kubernetes
already,
and
I
should
be
navigating
over
here
and.
A
Not
probably
not
oh
yeah,
just
just
just
a
slight
delay,
so
the
idea
is
like
to
to
ping
to
do
dns
resolving
for
different
domains,
and
I
was
just
using
o11y.lav,
cncfio
and
gitlab.com
now
when
we're
inspecting
the
cncf
ping
and
it
works.
Hopefully,
yes,
so
we
are
getting
some
results
back
and
everything
is
just
fine
and
hopefully
my
terminal
doesn't
die.
A
A
A
A
A
A
A
Now,
when
I
want
to
move
from
like
chaos,
engineering
also
looking
more
into
like
monitoring
two
observability,
which
is
like
the
second
thing.
We
just
talked
a
lot
about
metrics
and
how
you
measure
them
and
you're
defining
slos,
but
looking
into
observability
it's
much
more
than
that,
you
might
be
having
logs
and
events.
A
There
is
like
distributed
tracing,
continuous
profiling,
error,
tracking
rum
agents,
a
real-time
user
agent,
you,
real-time
user
monitoring,
agents,
yeah
and
a
certain
shift
from
monolith
to
microservices,
and
this
is
quite
a
lot
to
learn
as
a
developer
and
really
to
understand.
It's
like
take
a
step
back,
breathe
and
maybe
start
looking
into
metrics
and
tracing
in
the
beginning,
traces
itself,
a
little
like
different
to
logs.
It's
a
spain.
It's
a
span
with
a
start
and
an
end
time.
A
A
You
need
to
bring
your
own
back
end
so,
whether
it
being
for
traces,
jaeger,
elasticsearch
click,
house
and
so
on
can
build
your
own
distribution,
like
I
think,
aws
announced
the
the
metrics
aws
hotel
distribution
sga
today
or
yesterday.
I
think
michael
hausenblast
tweeted
about
it,
and
it
allows
you
to
also
like
use
auto
instrumentation
for
specific
languages
and
sdk.
So
you
don't
need
to
go
deep
into
your
the
application
code.
A
I
think
the
java
sdk
provides
that
another
idea
is
to
dive
into
more
into
auto
instrument
stage,
auto
instrumentation
and
observability,
and
look
into
evpf,
which
I
personally
find
super
interesting,
and
it
would
be
also
interesting
whether
we
could
dive
into
using
that
as
a
source
for
slos,
but
also
using
it
as
a
source
for
combining
all
the
different
data.
We
are
collecting
and
signals
and
events
and
really
moving
forward
with
that.
A
I
can
talk
10
hours
about
this,
so
this
is
the
the
call
to
action
to
to
navigate
to
ebpf
dot
io,
for
example,
and
check
it
out
and
yeah
with
that,
shifting
left
or
left
shift
the
thing
or
what
I
want
you
to
remember
is
see
the
value
and
observability,
so
it's
also
providing
an
application
inside
for
everyone,
who's,
not
the
developer
or
the
author
of
the
code,
which
allows
you
to
also
find
problems
fast.
So
it's
like
the
problem
inside
the
code.
A
The
other
thing
is
like:
when
observability
is
there
for
everyone,
you
need
to
teach
your
teams,
you
need
to
do.
Onboarding,
probably
write
documentation,
how
it's
being
done,
how
about
instrumentation
works,
defining
the
service
level
objectives
and
the
alerts
in
ci
cd
thinking
about
merge,
requests
with
staging
environments,
going
deeper
into
alert
channels,
incident
management
and
so
on,
and
from
a
cloud
native
perspective,
it's
great
to
just
have
like
the
benefit
of
the
deployments.
You
can
do
auto
scaling.
A
You
can
learn
from
all
the
cncf
projects
now
starting
to
adopt,
for
example,
open
telemetry
and
like
getting
the
best
practices
directly
and
getting
inspired.
So
I
I
kept
looking
into
how
open
telemetry
is
now
in
kubernetes,
in
prometheus
and
everywhere
else,
and
potentially
we
will
be
seeing
much
much
more
in
the
future
and
we
can
all
learn
together
from
from
the
amazing
open
source
community.
A
A
Potentially
vendors
are
already
working
on
that
chaos
out
of
the
box,
so
we
don't
need
to
like
add
it
on
top
of
it,
but
really
have
it
like
on
a
platform
or
something
like
that
and
should
be
accessible
to
everyone,
not
just
the
developer,
who
gets
everything
and
then
burns
out
from
that,
but
really
it
should
be
team
effort
and
yeah
telematics
open,
telemetry
being
adopted
more
widely.
I
started
something
for
myself
with
ci
cd
observability
with
regards
to
adding
open
telemetry
into
gitlab
different
story.
A
A
Do
the
shift
left,
try,
chaos,
engineering
and
benefit
from
observability
everywhere
yeah,
and
if
you
want
to
learn
more
about
observability,
I've
started
this
a
small
learning
platform
over
11
y
dot
love.
It
really
needs
like
everyone
to
contribute,
but
yeah.
I
really
want
to
like
encourage
everyone
to
learn,
and
I'm
also
here
for
you.
If
you
have
questions
around
epi
instrumentation,
I'm
really
I'm
overwhelmed.
I
was
overwhelmed
myself,
but
I
really
want
want
to
encourage
everyone.