►
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2023 in Amsterdam, The Netherlands from April 17-21. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Confidence with Chaos for Your Kubernetes Observability - Michael Friedrich, GitLab
The Kubernetes observability stack is deployed with Prometheus, and dashboards provide many interesting insights. What’s next? Everything is overwhelming and your teams are drowning in alerts. Alerts and Service Level Objectives (SLO) require team discussions and adjuments, and the dashboards could benefit from more fine granular details. Documentation and action items for your SRE and DevOps teams are needed too. Simulate a production incident to see whether SLOs are met, or alerts are fired. Is there a way to observe the deployed application, and see if it breaks from chaos? Join this talk to dive into ops and dev stories with practical insights into Kubernetes metrics, Prometheus alerting, chaos engineering with Chaos Mesh, OpenTelemetry app instrumentation, and hear about production incidents with failed SLOs. Gain confidence with chaos as an SRE, and as a developer seeing the value in Observability. Welcome to day 2 DevOps.
A
I'm
a
senior
developer
evangelist
at
gitlab,
and
if
you
want
to
like
find
me
online
I'm,
DNS
Miki,
which
is
DNS
mischi,
it's
the
lovely
version
of
Martin
and
German.
But
today
it's
not
about
me
today.
I
want
to
talk
about
what
is
like
observability.
How
can
we
gain
confidence
with
introducing
some
chaos
and
everything
around
like
what?
What
can
we
do
with
kubernetes
around
things
like
that?
A
A
We
have
potentially
kubernetes
up
and
running
and
we
need
to
kind
of
understand
what
we
should
be
monitoring.
There
are
like
the
different
components.
We
have
node
spots,
containers,
deployment,
Services,
apis,
ports,
data
sources
and
different
things.
I
don't
even
know
yet,
probably,
and
then
someone
says
well,
we
need
monitoring,
because
developers
cannot
work.
Deployments
are
broken.
Something
is
like
what
should
we
be
doing
there?
Is
it
like
availability
monitoring?
Should
we
be
looking
into
the
performance
and
resources
identifying
this
lower
or
the
blocking
deployments,
the
classic
traditional
service?
A
Monitoring
things
don't
really
work?
We
have
metrics
logs,
maybe
even
more
than
that,
do
we
need
to
understand
everything
to
observe
or
monitor
it
and
what
are
the
best
practices,
so
it
can
be
pretty
much
overwhelming
when
you
start
the
first
time
and
it
needs
focus
at
some
point.
So
in
the
first
iteration,
let's
look
at
metrics
within
kubernetes,
we
have
different
data
sources,
we
have
service
Discovery.
A
Luckily,
in
the
cncf
ecosystem,
we
have
Prometheus
with
the
slash,
Matrix
and
point
being
provided
being
scraped.
A
Time
series
database
built
in
we
can
calculate
Trends
use
dashboards,
have
service
level
objectives
and
from
there
go
into
alerting,
incidentsing
and
fixing,
and
coming
back
to
everything
for
this
like
thought,
this
is
like
a
huge
architecture
of
Prometheus
I.
Think
the
nice
way
is
like
integrating
with
the
services
career
within
qrators.
What
you
can
use-
and
everything
else
is
like
on
top
of
that.
A
There
is
a
lot
to
learn
so
I'm
also
I've
also
added
links
on
my
slides,
which
are
already
available
on
my
website.
So
you
can
look
it
up
later
on
async,
similar
to
how
to
visualize
the
data
within
premises
like
the
UI,
which
comes
out
of
the
box,
you
can
do
dashboards
in
grafana,
for
example,
and
I've
also
seen
that
there's
a
new
dashboard
as
code
and
GitHub
style
project
in
development
at
the
moment,
which
is
called
pulses
or
purses.
A
The
other
thing
is
like:
when
you
see
something
we
have
again
a
lot
of
things
to
to
analyze,
and
is
it
now
green?
Is
it
okay?
Is
it
healthy?
What
is
going
on
so
it's
nice
to
have
all
these
things
like
no
status,
resource
usage
deployments
and
and
different
other
things,
but
it
can
be
a
little
too
much
looking
at
a
lot
of
data
right
now,
right
now,
so,
for
example,
container
metrics
different
Integrations
should
I
be
looking
at
the
system,
CPU
usage
or
maybe
even
the
Pod
memory
usage.
A
Another
example
is
getting
the
health
of
the
deployments
with
Cube
State
metrics
different
ways
to
actually
do
that
and
use
that
and
I
think
the
nice
way
it
comes
out
of
the
box,
but
we
really
need
to
like
figure
out
hey
what
is
like
going
on
with
that.
So
the
idea
is
to
say
my
tricks
are
great
but
like,
what's
next
so
like
defining
the
definition
of
failure
like
the
what
then
do
something,
because
maybe
a
threshold
was
violated
or
a
specific
rule
was
matched
to
modify
and
raise
alerts.
A
The
who,
which
needs
to
be
the
responsible
person
and
team
and
going
to
the
hub,
which
can
be
like
documentation
for
incidents,
run
books,
corrective
actions
and
so
on,
which
also
means
reducing
the
mean
time
to
response
or
the
mean
time
to
resolve,
depending
on
the
definition,
and
it
got
me
thinking
like.
Okay
with
with
Prometheus,
we
can
integrate
the
alerts
with
Prometheus
alert
rules,
send
them
to
the
alert
manager.
I
have
some
like
grouping
inhibition
and
silences
different
transports.
A
That's
really
awesome
so
like
this
is
provided
out
of
the
box
for
kubernetes
itself.
I
also
found
like
a
website
which
is
called
awesome,
Prometheus
alerts,
dot,
grep.to,
which
also
provides
additional
alerting
rules
which
you
can
integrate
with
the
promises
operator
into
the
monitoring
I
of
of
the
kubernetes
cluster
and
get
these
alerts
ready
to
being
fired
now.
A
Integrating
them
into
the
premises
operator
can
be
done
with
the
premises
through
custom
resource
definition.
It's
just
an
example:
how
to
wrap
it
and
not
use
two
different
configuration
formats,
which
I
think
is
great,
especially
for
beginners,
don't
needing
to
learn
different
things
in
terms
of
an
alert
receiver.
Like
can
use
chat.
We
can
document
the
incidents
and
ticket
and
issue
systems,
maybe
even
mailing
lists,
but
at
some
point
it
can
be
overwhelming
and
the
so-called
alert
fatigue
can
come
up
and
saying:
I
have
10
000
alerts,
I
have
no
idea.
A
What's
going
on
and
basically
it's
like
I,
don't
Mark
everything
as
read
or
have
it
somewhere
and
the
calendar
goes
up,
and
maybe
we
need
sort
of
way
of
managing
these
incidents
in
a
better
way.
Now
the
idea
is
like
how
would
I
be
getting
an
alert,
just
like
the
the
visual
example
can
be
looking
at
the
alert
manager
in
premises
and
and
see
something
that
something
is
broken,
but
I
don't
want
to
do
that.
A
There
are
existing
chaos
Frameworks
already
as
open
source
projects.
They
allow
you
to
Define
experiments,
and
if
you
want
to
extend
everything
you
have
instrumentation
sdks
or
you
do
it.
The
German
Way
of
chaos,
engineering
on
the
right
hand,
side
no
just
kidding
one
example:
I
found-
is
chaos
mesh
or
one
open
source
project
in
the
CNC
landscape,
which
allows
you
to
fail
specific
things
in
a
kubernetes
cluster
or
on
hosts,
which
can
be
like
failing
apart
failing
the
network
and
seeing
how
the
application
is
behaving.
A
How
are
the
entire
deployments
break
HTTP,
not
even
like
the
responses,
but
didn't
check
some
headers
and
see
what's
what's
going
on
if
you
have
like
sort
of
some
scheduling
or
some
time
dependent
things
inside
breaking
time
and
ntps
is
also
a
good
idea
to
see.
What's
going
on
with
DNS
I.
Think,
probably
everything
is
a
DNS
problem,
but
yeah
it's
it's
interesting
to
see
when
you,
when
you
are
in
check
failures
with
DNS
or
just
random
responses
and
see
how
everything
is
going
nuts
or
maybe
even
continuing
to
work
running
chaos.
A
If
we
want
to
generate
some
chaos,
we
can
potentially
start
with
killing
some
ports,
and
this
is
a
nice
way
of
like
seeing
something
the
other
ways
like
when
the
port
ends
up
in
a
Crest
loop
back
off
kubernetes
tries
to
heal
itself.
So
is
this
like
the
correct
chaos
to
like
simulate
failure
in
a
cluster
and
with
like?
Maybe
we
can
find
something
else
so
thinking
of
more
real
world
example,
what
can
be
done
was
coming
back
to
DNS
in
in
in
the
previous
project.
A
This
was
like
six
years
ago
or
something
like
that.
We
had
an
application
which
was
creating
a
buffer
and
then
doing
some
DNS
resolving
and
then
doing
some
connections
and
when
everything
worked,
it
kind
of
close
the
connection
again,
and
the
thing
was
only
when
DNS
was
failing.
It
was
leaking
the
memory
above
so
like
every
second
one
megabyte
of
leaks,
but
only
when
DNS
is
failing.
A
We
as
developers
couldn't
really
reproduce
that
in
our
system
or
we
didn't
have
a
staging
environment,
we
only
had
our
death
machines
and
in
production
later
released
to
the
customers.
It
wasn't
really
fun
to
debug
really
really
late.
We
figured
out
hey,
it's
actually
DNS,
which
caused
that
problem
and
I
think
this
is
a
nice
way
to
really
like
figure
out
whether
everything
is
working
in
the
environment
and
the
idea
is
like
how
can
I
like
break
DNS.
A
A
And
these
are
basically
doing
not
much
but
trying
to
allocate
a
buffer
doing
the
DNS,
resolver
and
yeah.
Everything
else
should
be
just
looking.
Fine
we're
getting
some
IP
addresses
IPv6
even
and
it's
work,
so
the
other
thing
I
need
to
be
doing
is
accessing
the
chaos
mesh
dashboard
and
later
on,
Prometheus
and
blood
manager.
A
This
example
is
not
production
already,
because
it
will
break
everything
all
the
time
but
oops
another
good
idea.
I
should
be
submitting
at
the
bottom,
and
then
it
allows
me
to
verify
what
what
will
be
going
on,
so
it
again
shows
which
patterns
of
domains
will
be
affected
by
that,
and
I
can
also
see
a
preview
in
the
default
namespace,
for
example,
which
parts
will
be
affected
by
that
and
at
some
point
there
is
doing
too
many
demos,
the
open
observability
they
put
over
here
and
when
I
submit
this
and
submit
it.
A
A
I
need
to
cheat
I,
don't
know
the
from
Kill
query
at
the
top
of
my
head
and
we
want
to
see
that
as
a
graph.
So
right
now
memory
looks
okay,
it's
just
200k
something
like
that,
but
at
some
point
we
should
be
seeing
some
some
memory
going
on.
Let's
see
what
happens?
Okay,
so
host
not
found,
not
not
non-authoritative
try
again.
Data
means
we
are
already
injecting
DNS
failure.
A
So
at
some
point
we
should
already
be
leaking
some
memory
which
hopefully
yeah
it
goes
up
and
up
and
up
and
at
some
point
or
like
I've,
already
defined
a
premises
alert
for
that.
When
the
memory
usage
goes
beyond
10
megabytes,
10
megabytes,
it
will
trigger
an
alert
and
do
that
this
is
a
different
container
but
yeah.
Hopefully
this
will
work
in
the
meantime
Let's
see
we
have
the
alert
manager
over
here
and
some
point.
A
So
this
is
an
audio
alert
and
if
we
would
have
been
defining
some
service
level
objectives,
they
would
also
have
been
violated
in
that
regard.
So
this
can
be
like
one
way
of
injecting
chaos
and
seeing
that
everything
works
or
it
doesn't
the
the
demo
application
and
all
the
examples
are
available
online
as
open
source.
So
if
you
want
to
use
them
for
your
own
environments,
you
can
totally
do
that.
The
thing
I
want
to
do
now
or
like
continue
with
not
only
like
the
demon.
Now.
A
This
is
just
the
yellow
definition
again,
I'm
documented
for
you
to
maybe
try
it
later
on.
The
thing
is
now
that
we
have
generated
some
alerts
and
have
potentially
some
red
dashboards
or
something
maybe
optimize
or
think
about
optimizing.
The
alert
counts
think
about
grouping
additional
context
in
the
alerts
also
like
focusing
in
on
the
dashboard
so
like,
using
what
is
already
available,
correlating
data
and
also
often
reducing
the
amount
of
visible
data,
so
that
teams
who
are
like
you're
getting
paged,
potentially
a
3M
and
3,
am
in
the
morning.
A
You
really
need
to
see
what's
going
on
and
how
to
fix
it
fast,
because
otherwise
it's
really
not
fun
so
like
how
to
gain
confidence.
In
that
regard,
we
can
use
the
metrics
from
premises,
for
example,
Define
the
alerts
as
a
practical
query
and
have
the
service
level
objectives.
This
is
kind
of
a
step-by-step
storyline,
the
other
thing
which
is
important
to
notice
like
seeing
what
is
going
on
and
start
starting
with
the
golden
signals
like
latency
traffic
errors
into
saturation.
A
It
doesn't
really
matter
like
having
too
many
dashboards
and
alerts
like
learning
and
documenting,
and
also
ensuring
that
onboarding
for
new
team
members
Works
in
that
sense,
because
it
can
be
rather
overwhelming
and
the
goal
should
be
to
immediately
like
see
what
is
important
and
also
reduce
the
the
mean
time
to
response.
Overall.
A
If
you
want
to
go
into
like
customizing
Cube
Prometheus,
you
need
to
learn
jsonnet,
but
I
found
it
really,
like
rather
I,
would
say
easy
or
good,
documented
to
learn.
You
can
develop
your
own
rules
and
dashboards
and,
for
example,
by
monitoring
other
namespaces
adding
applications.
You
can
also
create
custom
dashboards,
adding
a
data
source
and
profana
for
Prometheus
at
the
panel
at
a
dashboard
and
even
automate
everything
and
not
having
to
create
everything
by
hand.
A
The
other
thing
which
is
great
is
the
service,
monitor
customer
resource
definition
provided
by
the
promises
operator,
which
allows
you
to
really
like
monitor
your
own
applications
and
existing
applications,
also
with
the
permit
and
with
the
metrics
endpoint.
So
the
tldr
basically
is
like
deploying
an
application.
It
has
a
matrix,
endpoint
just
adds
the
service
monitors
here,
the
into
the
deployments
and
get
even
more
observability
data
out
of
the
box,
but
this
was
a
lot
about
metrics
and
observability
is
more
than
that
and
we've
already
heard
today.
A
It's
like
six
types
of
events
or
it's
three
or
it's
even
more
than
that.
Potentially
we
will
be
seeing
a
lot
more
types
in
the
future
and
this
is
observability
kind
of
it's
like
collecting
all
the
data.
So
you
can
ask
the
questions
like
detecting
the
unknown
unknowns
for
logs.
We
have
a
lot
of
decisions
being
made,
so
it's
really
hard
to
answer
the
question
of
what
type
of
things
do
we
want
to
see?
How
long
do
we
want
to
store
it?
A
Is
it
helpful
to
see
all
the
podlocks
in
a
central
location
and
store
petabytes
of
data
with
retention
or
a
time
of
one
year,
because
compliance
and
things
like
that?
So
it's
it's
really
rough
with
tracing.
We
get
a
different
perspective
of
like
adding
spans
with
a
stunning
end
time
more
context,
more
metadata.
The
problem
on
the
other
side
is
you
need
to
do
code.
Changes
developers
need
to
be
adding
that
to
the
to
the
code.
Auto
instrumentation
is
another
thing
coming
up,
which
makes
it
easier,
but
still
it's
it's
challenging.
A
So
this
is
a
great
way
to
like
look
at
the
different
observability
data
type,
and
one
thing
I
recently
did
was
trying
out
the
open,
Telemetry
web
server
SDK
to
instrument
nginx
and
in
the
Pache
like
to
see
when
a
client
is
sending
an
HTTP
request.
The
server
as
the
backend
does
something
and
sends
the
kind
response,
and
it
got
me
thinking
of
like
can
we
maybe
add
a
chaos
experiment
to
that.
So,
like
slowing
down
the
response
and
I
could
add
a
sleep
in
my
code
and
then
deploy
and
then
see
what
what
happens.
A
But
this
isn't
really
like
the
best
idea
and
a
better
idea
would
be
to
use
a
chaos
experiment
for
HTTP
for
Network
for
CPU
and
memory
stress
tests
to
allow
seeing
what
is
going
on
and
I've,
reproduced,
like
one
of
the
first
things
with
chaos
mesh
to
stress,
test,
the
CPU
and
the
memory
forward
for
deployed
engine
X
container,
which
then
sends
the
traces
to
Jaeger
and
the
request
time
increased
over
time.
I
was
like,
oh,
this
is
for
me,
like
the
five
minute
success
to
get
going
and
and
to
really
like
think
about
more.
A
Think
of
like
how
can
we
trigger
that
with
chaos,
engineering
and
other
things,
combined
with
kubernetes
system
component
tracing
okay,
lots
of
things,
but
there
is
even
more
than
that
and
I'm
currently
trying
to
learn,
ebpf
and
understanding
what
it
actually
does
and
how
it
can
help.
There
were
great
ways
to
really
think
of
on
a
different
way
to
collect
observability
data
and
great
tools
to
getting
started.
But
one
of
the
ideas
was
like
how
how
does
this
like
complement
or
fit
within
Prometheus
metrics?
How
does
it
fit
within
traces?
A
Can
we
do
like
the
same
like
different,
defining
a
service
level,
objective
alerts
and
then
add
some
chaos,
engineering
and
I've?
Also
like
seen,
silium
tetragon
was
open
sourced
at
kubecon,
Europe
and
I
was
like.
We
should
be
combining
that
somehow
it
needs
an
open
standard.
It
needs
a
way
to
like
say
a
different
observability
data
types,
since
my
talk
is
too
long
already.
I
will
be
looking
into
this
next
year
when
I'm,
hoping
to
like
find
answers
together
and
learn
together
in
the
open.
A
So
there
are
different
types
of
chaos
or
chaos,
experiments
for
like
SRV
death,
devops
and
maybe
even
death
setups
like
overloading
CPU
failing
DNS
clients
that
are
not
closing.
Tls
connections,
for
example,
container
pools
not
being
successful
because
there
are
rate
limits
and
even
like
breaking
security
policies,
so
this
is
like.
A
A
What
I
mean
by
that
is
like
don't
run
everything
of
the
chaos
experiments
all
the
time,
because
it
could
break
existing
workflows
or
team
pages,
and
things
like
that,
and
maybe
think
of
like
starting
with
a
staging
environment,
to
prevent
data
loss
because
running
a
chaos.
Experiment
could
also
cause
like
at
a
certain
point,
maybe
another
database
right,
but
then
something
is
really
going
wrong
and
also
think
of
that
chaos.
A
B
D
A
A
This
first
time
experience
of
getting
started
and
kind
of
feeling
addicted,
and
the
thing
to
mention
is
it's
like
still
like
being
rude
or
on
the
open
heart
kind
of
so
you
should
be
using
our
bug
and
SSO
login
for
chaos
mesh,
which
you
can
configure
to
ensure
that
not
everyone
can
run
a
chaos
experiment,
because
then
it's
like
I'm
breaking
something
because
I'm
funny
or
maybe
I'm
a
malicious
attacker
or
something
so
really
may
also
making
sure
that
the
security
for
chaos.
Engineering
is
is
in
place.
C
C
E
Yeah
I
was
just
wondering
with
chaos.
Engineering.
Do
you
regularly
just
run
it
as
part
of
a
pipeline
like
what's
your
process
for
doing
or
executing
chaos
engineering?
Is
it
just
for
like
SRE
disaster
scenarios,
or
do
you
run
it
on
a
regular
basis?
Do
you
run
it
on
each
deployment
like
how
do
you
Implement
that,
as
part
of
your
workflow,
on
a
regular
basis,.
A
Say
like
there
are
different
use
cases
and
I
know
that,
like
from
from
the
gitlab
perspective,
our
SUVs
are
looking
into
ways
of
running
chaos,
engineer
or
chaos,
tests
for
like
the
the
production
environment
on
on
the
SAS
platform,
but
I'm.
Also,
thinking
of
how
can
we
like
enable,
for
example,
developers
within
or
like
everyone
using
that
in
the
CI
CD
workflow,
the
thing
is
like
running:
it
continuously
needs
sort
of
documentation
needs
ways
to
like
what
are
we
doing
with
the
results
of
that
chaos,
experiment
or
workflows.
A
So
this
needs
to
have
like
a
defined
workflow.
If
that
is
not
possible,
because
for
some
reason
it
generates
too
many
costs
because
it
can
also
be
like.
You
have
10
000
dollar
Cloud
build
because
of
chaos.
Engineering.
This
is
something
you
should
prevent
at
all
cost.
So
it's
it
really
needs
some
like
time
to
try
it
out
and
figure
out
whether
it
fits
into
your
systems.
A
If
the
experiment,
for
example,
can
be
a
deployment
into
stitching
or
environment,
from
a
merge
request
to
pull
request,
you
see
ICD
and
githubs
and
whatnot.
This
can
be
helpful
if
it
takes
too
long.
If
it
needs
to
run
five
minutes
and
then
another
five
minutes
and
things
like
that,
maybe
do
it
more
on
a
canary
deployment
basis.
Maybe
a
different
branch,
not
not
the
release,
branch
and
move
it
out
of
the
regular
feature.