►
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
The dawn of the digital transformation has led to drastic changes and innovations when it comes to software development. Current systems design and deployment methodologies require new approaches in order for engineers to reason about running systems. What is required for today’s DevOps teams to achieve the level of observability they need for modern architectures such as cloud environments, microservices and containers? Additionally, what are the challenges, trends, and cultural changes for observability of modern architecture?
In this presentation, Humio’s Lead Customer Engineer, Grant Schofield, will discuss the historical and current trends in software development that have driven the modern approaches to reasoning about complex distributed systems, including the impact of innovative technology like time-series databases, and unique approaches to observability such as leveraging data to create actionable alerts in real-time and at scale, and managing increased complexities instantly.
A
Alright,
let's
go
ahead
and
get
started.
I'd
like
to
thank
everyone.
Who's
joining
us
today
welcome
to
CNCs
webinar
building
the
culture
of
observability
within
your
organization,
I'm,
Kaitlin,
Barnard
marketing
manager
at
CNCs
and
I'll
be
helping
to
moderate
today's
webinar
I'd
also
like
to
welcome
our
presenter
today,
grant
Schofield
from
Cameo.
So
just
a
few
housekeeping
items
I'd
like
to
cover
before
we
get
started
during
the
webinar
you're,
not
able
to
speak
as
an
attendee.
So
there
is
a
Q&A
box
at
the
bottom
of
your
screen.
A
If
you
have
any
questions
throughout
the
webinar,
please
feel
free
to
drop
them
in
there
and
we'll
get
to
as
many
of
those
as
we
can.
At
the
end.
The
session
is
also
being
recorded
and
will
be
sent
out
within
24
hours
of
long
with
the
link
to
the
presentation.
So
with
that
I'll
hand
it
over
to
grant
to
kick
off
today's
presentation.
B
B
I've
been
assisted
men
for
almost
you
know,
20
years
and
been
on
the
internet,
since
I
was
13
and
using
Linux,
and
during
that
time,
I've
seen
a
really
big
change
in
how
we
build
and
ship
software,
but
also
how
we've
run
it
and
how
we
manage
it,
how
we
monitor
it
and
make
sure
it
stays
up
previous
to
humo,
where
I
work.
Now,
where
I'm
the
director
of
infrastructure
I,
was
the
director
of
infrastructure
at
vivo,
the
music
video
company
during
those
years
I
spent
around
three
and
a
half
years
there.
B
We
were
early
adopters
of
many
cloud
native
technologies,
so
we
were
big
kubernetes
users
we
implemented
distributed
tracing.
We
had
log
aggregation
and
we
had
everything
else.
Metrics
and
monitoring.
So
I
learned
a
lot
about
the
changing
ways.
Were
shipping
software
and
monitoring
it
in
these
very
complex,
distributed
systems
that
were
building
today.
B
What's
going
to
change
in
the
way
that
we
make
systems
observable.
So
during
this
whole
presentation,
I'm
going
to
be
talking
about
how
you
can
make
systems
of
Virk
observable
within
your
organization
and
these
systems
just
aren't
your
servers
and
your
applications,
but
there's
actually
broader
applications
of
the
observability
within
your
organization
and
alignment
that
you
can
bring
from
the
business
side
and
the
engineering
side
that
benefits
the
users
and
that's
a
key
thing
that
I'm
also
going
to
talk
about
is
the
focus
on
users
with
an
observability.
B
So,
first,
what
observability
isn't.
So
it's
actually
not
a
new
concept.
It's
an
it's.
An
older
mathematical
concept
that
was
used
and
we're
going
to
talk
about
that
observability,
isn't
monitoring
monitoring
is,
is
really
just
the
systems
that
are
alerting
me
when
something
goes
wrong
and
that
isn't
really
observability.
It
goes
much
further
than
that
and
it's
just
not
your
operations
team's
concern.
B
Everyone
in
your
organization
should
care
about
observability,
specifically
in
engineering
just
like
DevOps,
so
culturally,
it's
really
very
similar
to
DevOps
so
observability
as
its
discussed
in
the
community.
These
days,
I
took
this.
You
synthesize
this
from
a
few
different
places,
but
it's
the
intersection
of
systems,
applications
and
users.
B
B
Even
an
individual
user
is
really
what
observability
is
to
me
and
how
can
we
tailor
our
systems
to
get
that
result
and
the
the
way
that
I
also
describe
it
to
to
others
is
when
I'm
looking
for
that
needle
in
the
haystack
observability
gives
me
the
context
to
know
why
that
needle
is
pointing
a
certain
way.
Why
is
it
oriented?
B
Why
is
it
in
this
layer
of
the
haystack
and
things
like
that,
and
it
really
allows
me
to
understand
all
the
attributes
of
that
needle
and
at
the
end
of
the
day,
this
is
what
I
need
is
an
engineer
to
debug
problems
effectively
with
minimal
steps.
Do
I
have
the
information
I
need
about
the
problem
I'm
looking
at
to
solve
the
problem?
B
Most
importantly,
observability
needs
to
be
a
cultural
facet
of
your
organization.
Just
like
DevOps
has
become
today.
It's
the
responsibility
of
everyone
in
the
organization
to
care
about
observability,
and
this
this
means
from
the
Cee
CEO
on
down,
because
this
data
that
we're
generating
with
our
observability
systems
can
actually
benefit
the
entire
organization
via
marketing,
finance,
etc,
and
observability
is
going
to
be
different
for
every
organization
because
what
your
users
are
concerned
about
and
what
your
engineers
are
concerned
about
is
going
to
be
different
than
everyone.
B
B
So
the
original
idea
of
control
theory
came
from
systems
engineering,
and
so
the
goal
was
is
if
we
have
a
complex
system
and
usually
we're
talking
about
mechanical
systems
or
electrical
systems,
if
I
put
in
an
input,
can
I
judge
what's
going
to
come
out
of
that?
So
observability
came
originally
from
control
theory.
B
It's
a
sub
domain
of
control
theory,
one
aspect
of
this
subfield
of
mathematics,
it's
created
by
Rudolph
camp
common
and
in
control
in
control,
Terry
were
trying
to
judge
the
internal
state
of
a
system
based
on
external
outputs
and
the
observability
is
the
ability
to
look
at
this.
So
Commons
work
was
really
important
in
the
50s
and
60s,
in
particular,
with
NASA
on
the
Apollo
missions
and
the
shuttle
programs
in
order
to
develop
reliable
systems
and
test
those
systems
before
it
went
up
into
space.
B
B
So
it
was
very
formative
at
the
time
and
it
still
is
impacting
us
today
and
it's
an
interesting
it's
an
interesting
idea
that
observability
was
the
word
chosen
by
everyone,
but
it
really
makes
sense
in
terms
of
what
the
system
is
and
what
the
theory
is
in
terms
of
how
it's
impacting
how
we
talk
about
it
with
an
infrastructure
so
observe.
The
observing
systems
of
the
past
was
quite
different.
Software
as
a
service
didn't
exist,
so
we
built
everything
and
when
I
say
we
built
everything,
sometimes
we
were
using
something
off
the
shelf
like
before.
B
Now,
yes,
there
was
net
saying
or
Big
Brother
there
were
there
were
these
systems
that
we
could
put
together,
but
we
were
generally
having
to
assemble
them
ourselves
and
provide
the
data
and
unique
ways
right
our
own
systems
to
query
our
databases
or
look
at
our
processes
and
our
memory
usage.
All
these
things
were
generally
done
by
hand
and
bespoke
for
the
for
the
environment.
You
are
working
in.
B
There
were
expensive
and
ugly
tools
like
HP,
openview
and
various
CA
tools
to
look
at
SNMP
and
things
like
that,
but
at
least
being
a
being
in
Linux
environments
since
the
since
the
late
90s.
That
was
never
something
you
really
had
access
to.
Maybe
our
switches
had
SNMP,
but
we
weren't,
we
weren't,
really
generally
spending
money
on
tools
like
HP
openview,
to
manage
those
systems.
So
we
scripted
everything
or
we
died.
Trying
I've
done
lots
of
questionable,
Perl
and
bash
in
my
past
and
we
ran
it
all
with
cron.
B
Everything
was
running
the
Acron,
and
hopefully
you
know
everyone
got
bit
because
you
know,
hopefully
everyone
you
know
learned
the
less
and
that
sometimes
cron
jobs
lasts
too
long,
and
then
you
can
build
up
a
kind
of
crown
jobs
and
take
down
your
system
just
with
cron
jobs,
because
it's
too
busy
connecting
to
different
systems
and
something's
happening.
So
it
generally
wasn't
as
reliable
as
what
we
have
today
and
in
general,
we
were
are
syncing
things
around
all
of
the
time.
B
So
we're
are
thinking
around
configure
our
thinking
around
blogs,
and
things
like
this
is
very
a
synchronous.
At
that
time,
so
what
are
some
of
the
things
that
we
built?
This
was
kind
of
fun,
going
back
to
the
past
and
looking
at
these,
these
systems
that
we
used
to
write
now.
This
isn't
mine
I,
had
to
go
on
the
internet
and
find
this
one.
But
this
is
what
a
basic
dashboard
would
look
like.
B
One
of
the
things
that
I
learned
when
running
Big
Brother
is
that
red
and
green
being
good
and
bad
isn't
necessarily
the
best
combination,
because
there
are
a
good
amount
of
people
in
the
world
that
are
colorblind.
So
it's
one
other
thing,
one
of
the
first
lessons
I
learned
about
making
observable
system,
is
you
actually
have
to
care
about
the
UI
and
how
it's
going
to
affect
people
and
how
are
they're
going
to
use
it
now
granted
to
you?
B
B
So
when
we
talk
about
distributed
systems
today
and
how
we
monitor,
we
were
doing
very
you
know
basic
you
know
takes
on
this
20
years
ago
with
Big
Brother,
it's
a
distributed
system
and
we're
using
one
system
to
query
systems
and
piping
that
to
other
systems,
and
then
one
thing
that
I
always
remember-
that's
that's
fun.
To
think
about
is
we
used
to
have
to
have
modems
in
the
data
center?
B
The
internet,
while
it's
far
more
reliable
today,
was
pretty
questionable
back
then
at
times
and
and
when
you're
running
a
production
system
that
people
are
depending
on.
So
at
this
time,
I
was
running
a
airline
reservation
system
for
iha
software
in
a
data
center
we
needed
to
know
when
things
went
down
and
couldn't
trust
the
internet,
so
we
actually
had
modems
that
were
dialing
text
pagers
to
actually
page
us,
because
that
was
the
only
reliable
way
to
do
it
at
the
time.
So
things
have
come
a
long
way.
B
This
is
one
of
the
first
things
that
I
implemented.
That
I
was
really
excited
about
that
I
thought
was
going
to
solve
all
my
problems
for
these
via
VRML
diagrams
of
of
your
systems
and
the
health
of
the
systems
and
how
they
were
connected.
But
at
the
end
of
the
day
was
it
was
a
pretty
trick
that
didn't
really
solve
the
problems.
It
didn't
really
make
the
systems
more
observable,
but
it
gave
me
a
little
bit
better
of
a
picture
of
things.
B
Things
generally
were
based
on
Rd
tool
if
you're
familiar
with
Rd
tool,
it's
a
round-robin
database.
So
what
would
happen
is
you
would
have
really
a
lot
of
data
to
look
at
for
a
day
and
then,
as
time
went
on,
you
would
essentially
roll
up
the
data,
and
so
I
had
a
six
month
view,
but
it
didn't
have
the
the
amount
of
data,
so
you
definitely
could
lose
outliers
and
things
like
that.
We
were
using
graphs
more
for
in
the
immediate
problem.
Solving
not
necessarily
looking
back
on
our
past
problems.
B
Things
got
a
little
bit
better
with
cacti
things
looked
better,
but
it
was
still
a
fairly
asynchronous
we're
generating
these
graphs.
It's
very
static,
so
I'm
generally
generally
using
a
script
to
generate
a
list
of
hosts
to
make
these
graphs
or
arsenic
data
around
and
things
like
that.
It
really
wasn't
fluid
and
we
didn't
really
have
much
configuration
manage
and
at
the
time.
So
were
these
systems
observable
yes
kind
of
I,
don't
think
they
necessarily
meet
the
demands
of
today
or
how
we
were
looking
observability.
B
B
I
think
a
lot
of
the
work
and
observability
the
last
several
years
has
been
about
taking
this
intuition
that
a
lot
of
operators
have
and
putting
it
into
software,
putting
it
into
code
to
show
us
problem
so
I
think
our
systems
are
observable
because
we
there's
how
our
systems
were
put
together,
but
in
general
that's
how
we
operate
it.
There
are
a
lot
of
limiting
factors
back
then,
in
terms
of
why
we
did
things.
The
way
that
we
did
them
and
cost
of
storage
and
compute
was
was
a
lot
different.
B
We
were
grabbing
a
lot
of
logs
generally.
You
would
find
a
problem
in
the
graph
and
you're
logging
into
a
machine
and
you're
tailing,
a
log
or
grepping
a
log
to
count
errors
or
things
like
that
or
to
distinguish
the
errors
from
not,
and
there
were
no
time
series
databases
and
there
was
no
no
sequel
so
we're
relying
on
file
systems
and
databases
and
like
my
sequel
at
the
time,
and
because
the
cost
of
storage
and
compute
was
so
high,
didn't
necessarily
always
make
sense
to
use
use
those
resources
for
your
monitoring.
B
So
file
systems
are
generally
the
lowest
common
denominator
of
how
we
worked
with
the
data
and
how
we
moved
it
around
because
of
things
like
Rd.
We
didn't
really
have
context
of
the
past,
so
hopefully,
when
there
was
a
problem-
and
you
wanted
to
record
I
had
this
problem
and
here's
how
I
fix
it,
it
was
generally
zipping
up
gzipping
up
a
bunch
of
logs.
B
Maybe
some
directories
of
graphs
and
saying
here
was
this
problem:
here's
how
I
solved
it
and
here's
the
graphs
afterwards
and
I
was
sharing
that
within
my
organization,
but
that
takes
a
lot
of
people
overhead
in
order
to
do
that,
and
you
have
to
have
a
group
where
everyone's
bought
into
that,
if
only
one
person
is
doing
it's
not
nearly
as
effective
as
if
everyone
is
doing
it,
some
of
the
other
limiting
factors
I
think
in
how
we
think
about
observability.
In
the
passes
there
was
always
a
sysadmin
problem.
Yeah
we
call
it.
B
We
don't
call
them
sis
admins
anymore,
but
it
was
always.
Oh,
the
operations
groups
problem
to
handle
these
things.
We
didn't
really
have
a
lot
of
configuration
management.
You
know
there
were:
there
were
a
few
things
out
there
at
the
time,
but
in
general,
though,
we
didn't
even
have
distributed
source
control,
we
had
CVS
and
things
like
that.
So
the
modern,
the
modern
takes
on
configuration
management,
rely
on
just
distributed
source
control
and
we
have
all
those
things
now.
B
So
it's
much
easier
today
and
generally,
they
think
that
a
lot
of
the
technologies
that
were
dealing
with
were
for
lack
of
a
better
term,
Enterprise
II,
and
so
they
were
coming
on
a
Cisco
and
they
were
coming
out
of
HP
and
they
were
really
tailored
for
those
customers.
People
working
more
in
the
open
source
world
were
generally
integrating
in
these
proprietary
systems,
and
SNMP
was
a
very,
very
complex
one
that
you
know
have
all
these
and
MIM
files
and
things
like
that
and
it
was.
B
It
was
more
for
recording
your
switch
and
router
data
and
less
less
for
other
things
unless
you're
running
something
like
a
Solaris
or
maybe
yeah
hp-ux.
So
most
of
the
data
that
we're
generating
was
statically
generated,
data
from
logs
and
the
tools
were
a
very
macro
view
of
the
world.
We
didn't
have
that
ability
to
peer
into
systems
like
we
do
today
and
with
the
way
that
we
run
systems
today.
B
The
approach
has
to
be
a
lot
different
due
to
the
complexities,
so
the
sysadmin
squad
goals
back
in
the
past,
we're
not
letting
engineers
access,
production
and
change
things
out
from
under.
You.
I
think
that
since
the
advent
of
DevOps
and
the
change
in
culture
and
the
change
the
way
that
we
ship
software,
this
has
definitely
changed.
But
that
was
definitely
the
norm
back
then,
having
monitoring
at
all
was
fairly
forward-thinking
at
the
time
it
seems
there
were.
There
were
definitely
definitely
organizations
that
didn't
have
any
monitoring
and
still
today,
I've
talked
with
organization.
B
George
is
now
starting
to
monitor
their
system
or
all
their
monitoring
is
very
reactive.
So
getting
alerted
reliably
was
a
problem
as
I
mentioned
in
the
past,
and
so
he
had
to
build
things
in
order
to
do
that
as
well.
Uptime
was
the
most
important
thing,
I
think
and
I'll
discuss
this
a
little
bit
later
up
times
a
little
bit
easier
to
come
by
it
today
than
it
wasn't
a
pass.
But
everyone
was
obsessed
with
five
nights
in
late
90s
and
early
2000s,
and
that's
that's
really
what
we
were
working
for.
B
I,
don't
think
that
was
necessarily
the
right
focus
either
as
I'll
discuss
later
and
cross-training,
and
teaching
in
tuition
was
a
very
important
thing
operations
group
in
the
past,
because
that's
how
we
were
able
to
reason
about
the
complex
distributed
systems
so
fast
forward
to
the
present.
So
we're,
basically,
you
know
looking
you
know,
10
15
years
out
from
the
I
think
from
when
Linux
really
got
into
the
data
center
or
using
it
heavily.
So
what's
changed
in
those
years.
So
a
lot
has
changed.
Of
course,
fifteen
years
of
Moore's
law.
B
We
have
all
these
different
technologies
that
have
come
along
in
response
to
how
we're
building
and
shipping
software,
but
all
that
relies
on
the
newer
hardware
and
things
like
SSD.
That
makes
certain
things
possible
that
weren't
possible
in
the
past,
so
a
lot
has
changed,
and
so
how
do
we
fit
observability
in
with
this
picture
that
we
have
today,
since
it's
quite
different,
so
the
key
factors
driving
observability
today,
one
of
the
major
ones,
I,
think
that
no
one
talks
about
a
lot
is
the
dump
the
democratization
of
Technology.
B
So
when
we,
when
we
talk
about
containers,
arguably
it's
an
outgrowth
of
the
m's
and
if
you're
running
in
the
cloud
you're
running
on
beams,
you
know
as
it
is
so
all
of
us
are
dependant
on
VMs
and
that
technology
took
a
while
to
trickle
down,
in
particular
from
VMware.
In
particular,
there
are
other
people
as
well,
but
in
particular
via
VMware,
and
things
like
my
sequel,
took
a
long
time.
Things
like
my
sequel
and
Postgres
to
quote-unquote
catch-up
with
Oracle
to
have
things
like
materialized
views.
B
Some
of
that
is
due
to
that
technology,
trickling
down
from
the
more
enterprise
approaches,
so
the
trade-offs
are
much
different
today.
The
way
that
I'm
looking
in
my
system,
building
my
monitoring
system
or
my
observability
systems
are
a
lot
different,
because
the
systems
are
more
complex,
and
so
we
need
more.
We
need
more
systems
and
more
complex
approaches
to
manage
those.
B
So
many
organizations
are
quite
comfortable
spending,
a
lot
of
money
on
a
database
like
an
elastic
search
or
he
neo
in
order
to
store
their
log
data,
because
it's
the
most
effective
way
to
go
about
debugging
the
problems
today,
whereas
before
it
was
always
monitoring
and
these
metrics
were
the
domain
of
the
system
ends,
they
needed
to
figure
out
donating,
devoting
huge
resources
or
whole
teams
for
those
things
wasn't
really
practical
at
the
time,
but
that's
changing
today.
So
monitoring
is
about
predicting
failure.
B
When
I'm,
when
I'm
deploying
my
monitoring
I'm
doing
things
like
is,
you
know,
did
is
this
system
up
or
down?
Do
I
have
a
high
number
of
errors,
but
all
these
things
are
me
forward.
Thinking,
I
this
and
setting
up
strict,
or
you
know,
setting
up
monitors
to
react
to
specific
situations
that
I'm
predicting
systems
fail
a
lot
differently
today,
because
they're
more
complex-
it's
not
just
you
know
the
basic
sense
of
proxy
and
a
bunch
of
application
servers
in
the
area
of
micro
services.
B
These
things
are
more
complex,
we're
using
third-party
services
in
an
Amazon
or
a
GC,
p4
databases
or
things
like
Redis
or
I'm,
using
a
third
party
API
to
do
face,
detection
or
things
like
that.
So
we
have
much
more
complex
systems
and
it's
hard
to
write
your
monitoring
correctly.
To
take
account
for
all
of
these
complexities.
B
The
world
that
we
live
in
today
is
far
more
ephemeral,
especially
for
deploying
on
the
cloud,
but
I
would
say
that
even
within
the
data
center,
the
approach
is
moving
more
towards
ephemeral
systems.
The
days
of
configuring
configuration
management,
setting
up
a
server
and
configuring
your
server
and
that
server
is
going
to
stay.
The
way
it
is
are
fairly
long
gone
just
because
the
nature
of
the
systems
are
running
today
and
the
cost
efficiencies
we
get
through.
B
Having
a
flexible
architecture,
micro-services
have
really
influenced
I
think
the
way
that
we
look
at
observability,
because
when
you
start
deconstructing
your
services
into
several
micro
services,
you
end
up
getting
these
dependencies
between
the
two
and
in
order
to
debug
your
system
or
make
your
system
observable
and
understand
it.
You
have
to
understand
how
those
micro
services
are
working
together
if
there's
a
problem
with
one,
if
there's
latency
and
things
like
that,
so
micro
services
are
really
driving
that
as
well.
B
More
and
more
companies
today
are
looking
to
have
a
multi
region
or
a
multi
cloud,
a
multi
cloud
set
up,
and
this
creates
a
whole
nother
level
of
of
complexity.
In
terms
of
in
order
to
reason
about
my
system,
I'm
now
having
to
go
outside
of
my
data
center
to
understand
things
and
those
things
are
things
like
BGP,
anycast
or
DNS
or
CD
ends,
and
all
these
different
technologies.
That
I
also
have
to
make
observable
to
understand
how
my
how
my
services
and
the
data
center
are
working.
B
The
complexity
is
increased
exponentially
talk
a
lot
about
this
in
terms
of
the
changes
that
we've
had
monitoring
and
metric
systems
have
come
a
long
way
in
terms
of
in
the
past
generally.
The
way
that
we
would
configure
them
is
we
have
these
static
host
lists,
so
I
would
generally
loose
a
post
running
my
web
servers
and
my
check
would
check
those
web
servers
or
ping.
My
web
servers
and
tell
me
if
there
it
was
timing
out
or
tell
me
if
it
couldn't
connect
and
things
like
that.
B
Well,
with
the
way
that
we
run
systems
now,
especially
with
docker
and
things
being
ephemeral,
our
monitoring
system
has
to
understand
our
environment
far
more
than
it
ever
did.
So
it
has
to
be
querying
the
kubernetes
api
and
getting
the
list
of
pods
and
things
like
that,
or
it
has
to
be
built
in
running
far
more
disparate
systems,
especially
with
the
use
of
external
api's
these
days,
and
so
we
we
also
have
to
understand.
B
So
when
talking
about
modern
observability
people
like
to
talk
about
there
being
four
pillars
of
observability
and
those
are
metrics
logging
distributed
tracing
and
then
some
people
say,
there's
three
I
like
to
include
the
fourth
I
think
it's
becoming
more
common.
When
we
talk
about
the
fourth
pillar,
being
the
user
experience
and
that
usually
comes
down
to
your
alerting,
visualization
or
quote-unquote
monitoring.
B
B
That's
that's
a
you
know:
nginx
500s
or
four
hundreds
in
order
to
use
that
data
on
bait
on
each
of
my
nginx
servers,
I'm
generally
tagging
that
data
and
I'm
tagging
that
data
with
maybe
a
host
or
you
know
maybe
a
process
ID
or
things
like
that,
and
so
what
I'm
actually
doing
it
just
just
or
the
data.
In
these
time
series
databases,
I'm
tagging
it
in
order
to
see
that
data
later,
but
that
can
cause
problems.
B
B
You,
you
IDs
or
any
you
IDs,
because
what
that
results
in
is
high
cardinality
and
the
high
cardinality
is
a
problem
when
you're
dealing
with
a
database
essentially
I'm
having
to
go
through
too
many
tags
and
process
too
many
tags
to
actually
get
up.
My
data
metrics
are
valuable
because
they
have
far
more
predictable
overhead
than
logs.
It's
it's
easier
to
scale
them
up
and
add
a
lot
more
than
it
is
to
add
a
lot
more
logs.
B
Metrics
are
far
usurer
today,
because
there's
lower
integration
costs-
and
this
is
a
result
somewhat
of
people
getting
used
to
APM
and
using
APM
systems
and
just
having
a
library
that
instruments
my
database
calls.
My
cache
calls
just
into
these
things
on
my
behalf
and
so
I
don't
have
to
worry
about
it.
So
it's
much
easier
to
get
metrics
out
of
the
system,
but
if
using
those
metrics,
then
it's
not
valuable
to
actually
save
them
and
you
don't
always
know
which
metrics
you
need-
and
this
goes
back
to
that
same
concept
about
monitoring.
B
If
I
wasn't
able
to
reason
about
them
and
that
worked
really
well,
when
we
were
still
deploying
systems
very
asynchronously.
But
in
today's
times
it's
it's
quite
different
in
those
those
logs
aren't
always
the
server's.
You
can't
when
your
doctor
goes
away,
you
need
to
make
sure
those
logs
are
somewhere.
So
I
think
you
know.
I've
evolved
my
thinking,
especially
since,
coming
into
the
kubernetes
world
and
see
logs
as
a
very
important
part
of
how
we
make
our
systems
observable.
B
So
in
general,
the
challenge
with
logs
is
that
they
are
the
least
standardized
in
most
environments
in
general,
most
most
engineering
orgs,
you
know
they
might
use
log4j
or
they
might
use.
You
know
insert
you
know
Winston
or
what
have
you
in
your
different
environments,
but
what
you're?
Actually
logging
can
be
quite
different
in
terms
of
what
one
group
cares
about?
B
They
might
change
the
fields
like
there's
generally
not
the
same
approach
on
a
lot
of
organizations,
even
if
they're
doing
structured
logs
like
Jason
the
fields
aren't
always
the
same,
for
instance,
so
generating
logs
is
the
easy
part
storing
it
is
not
and
storing.
It
is
a
hard
challenge
and
why
you
see
people
relying
more
on
no
sequel
type,
databases
like
an
elastic
search
or
human
in
order
to
store
their
logs,
and
you
can
only
really
store
a
finite
amount.
B
I
can
archive
them
up
to
a
three,
but
can
I
actually
access
them,
so
it's
it's
useful
to
archive
them
off
to
s3
to
go
look
at
them,
but
it's
very
difficult
to
imagine
even
several
years
from
now
a
storing
ten.
You
know
you
know
ten
years
of
logs
in
our
log
management
systems
and
querying
it
so
and
in
general
it's
a
good
idea
to
sample
your
logs
and
generate
metrics
from
them.
So
you
can
use
that
later.
If
you
do
need
to
reference
your
logs.
B
B
Tracing
is
the
newest
component
of
modern
infrastructure
and
I
would
say:
I,
don't
say:
oh
we're
still
learning
about
it
and
the
way
that
it's
progressed
over
the
last
several
years
is
really
interesting
to
me
in
terms
of
the
open
tracing
spec
and
everyone
getting
on
board
with
that
proprietary
software
is
service,
vendors
or
things
like
Zipkin
and
Jaeger.
It's
really.
It's
really
come
a
long
way,
however,
it's
quite
difficult
in
most
environments
to
make
it
useful
unless
everyone
is
using
it.
B
So
unless
you
have
distributed
tracing
integrated
into
your
entire
environment,
all
your
engineering
groups
are
using
it
or
using
something
like
sto,
more
envoy.
That
can
give
you
that
it's
there's
a
limited
amount
of
utility.
Unless
everyone
is
using
it
because
then
you
will
have
gaps
in
your
distributed
tracing,
so
a
good
place
to
start
is
actually
just
having
correlation
IDs
and
just
a
brief
talk
about
how
that
works.
B
At
the
very
least,
I
can
use
that
correlation
ID
to
see
how
a
request
flowed
through
my
system,
and
so
that's
generally
a
good
place
to
start.
If
you
can't
really
go
in
on
distributed,
tracing
I'd
like
to
point
out,
the
distribute
tracing
is
not
quite
APM.
They're
quite
different,
so
I
think
that
there's
a
lot
of
synergy
between
the
two
and
many
of
the
many
of
many
vendors
are
offering
distributed
tracing
along
with
their
EPM
products.
Now
but
they're
not
quite
the
same,
and
so
you
shouldn't
conflate
the
two
and
finally,
the
fourth
pillar.
B
Think
that
the
important
thing
here
for
me
is
that
alerting
and
monitoring
or
are
very
important.
But
that's
not
what
makes
your
system
observable
and
you
need
to
worry
about
alert
fatigue
and
making
those
actions
making
those
alerts
actionable.
Putting
developers
on
call
is
the
right
thing
to
do,
but
as
an
Operations
organization,
you
need
to
make
sure
that
those
alerts
are
manageable
and
include
the
proper
context,
and
this
is
the
important
part
about
building
observable
systems
that
allows
you
to
give
people
access
to
the
access
to
debugging.
B
This
information
really
fluidly
without
having
to
go
to
five
different
applications
and
and
make
sense
of
it
and
reason
about
it
themselves,
because
not
everyone
quite
has
that
intuition
about
those
systems
unless
they're
the
ones
actively.
Maintaining
it
I
think
that
the
most
important
part
about
bringing
a
good
standard
of
alerting
and
monitoring
into
your
environment
is
really
managing
it.
B
But
you
know
maybe
one
one
tweak
later
and
we're
not
really
sharing
that
information
with
with
the
entire
org.
When
we
do
that
so
today,
what
do
we
have
as
our
current
groundwork?
So
engineers
want
a
PMS,
a
PM
ish
things.
We
know
that,
and
so
that's
really
grown
out
of
the
DevOps
movement
and
I
think
developers
taking
more
of
an
interest
in
how
their
systems
are
performing.
The
rise
of
SAS
infrastructure
companies
has
really
changed
things,
be
the
data
dog
or
a
sumo
logic,
signal
effects,
etc,
etc.
B
We
have
elastic,
search
and
Elkin
software's
of
services,
and
he
me,
oh
of
course,
and
there's
far
more
options
in
the
past
than
there
were
for
all
of
these
systems
that
we're
talking
about
there's,
open
and
closed
source
options.
Open
tracing
has
had
a
really
big
impact
on
making
distributed
tracing
accessible
to
many
people.
B
We're
able
to
run
systems
of
high
throughput
ourselves,
it's
not
an
it's,
not
on
a
question
to
run
assist
in
yourself.
They
can
ingest
10,
20,
30
terabytes
of
logs,
given
proper
machines,
and
it's
not
that
hard
to
manage
it's
far
easier
than
it
used
to
be
in
the
past.
So
that
gives
us
the
ability
to
store
more
data,
be
it
more.
You
know
more
data
from
today
with
more
detail
or
data
further
into
the
past.
That's
far
either
easier
to
integrate
disparate
systems
today,
thanks
to
Jason.
B
B
The
important
part
is
this
is
having
empathy
for
that
user.
That
user
doesn't
necessarily
care
about
what
your
CPU
load
is,
or
your
memory
usage
or
how
many
containers
you
have
and
the
limits
on
those
containers,
those
those
aren't.
Your
users
concerns
what
they
care
about
is
did
I
get
an
error.
Did
my
login
fail
and
how
slow
is
this
site,
so
your
observability
infrastructure
should
really
focus
in
on
these
user
concerns.
Am
I
focusing
in
on
these
user
concerns?
You
can
tailor
your
observability
goals
to
those.
B
So
how
do
you
go
about
doing
that?
You
gather
as
much
context
as
you
can
from
the
path
of
the
user.
I
think
that
systems
are
far
more
observable
when
they're
in
our
data,
centers
or
they're,
in
our
kubernetes
clusters
or
in
our
cloud.
But
you
really
need
to
expand
your
observability
past
that
you
need
to
be
taking
the
logs
from
your
CD
ends.
B
You
know
your
load,
balancers,
etc
and
bringing
those
into
your
your
logging
system
and
your
metrics
generating
systems
and
I
think
you
should
even
take
this
further
in
terms
of
instrumenting
the
clients
to
focus
on
key
experiences.
Just
briefly,
one
of
the
things
that
we
did,
vivo
that
turned
out
to
be
really
powerful
was
we
wrote
around,
so
we
did
music
videos
and
we
had
our
own
video
player
and
in
that
video
player
we
in
instrumented
things
like.
How
long
did
it
take
that
video
to
start?
B
How
long
did
the
DNS
request
take
in
order
to
look
up
this
round
robin
DNS?
What
CDN
did
I
hit?
How
long
did
it
take
for
the
first
frame
to
start
and
all
of
those
all
of
those
things
that
were
useful
for
us
operationally
to
judge
how
the
systems
were
working
all
over
the
world?
In
fact,
it
helped
us
discover
problems
with
a
certain
provider
in
Brazil.
B
Had
you
know,
lookups
of
6
700
milliseconds
for
for
our
DNS,
which
is
quite
slow,
so
it
gave
us
really
interesting
data
about
specific
users,
and
you
need
to
remember
that
if
you
have
a
million
requests
a
day
and
10
percent
of
those
fail,
that's
you
know.
10,000
people,
that's
that's
elute
leo!
That's
a
lot
of
requests
to
fail,
so
these
are
things
that
are
worth
taking
the
time
to
implement
all
the
way
out
to
the
client.
So
you
really
understand
how
your
system
is
working.
B
So
this
is
why
you
need
to
develop
a
DevOps
approach,
your
organization,
when,
in
terms
of
observability,
so
it's
a
combination
of
you
know
not
just
you
know
your
infrastructure
team
and
your
application
engineers,
but
even
broader
than
that
into
your
business.
Your
financing,
your
marketing,
standardizing
your
engineering
organizations
requirements
for
what
people
are
logging,
what
they're,
creating
metrics
for
if
they
have
tracing
or
not,
will
help
a
lot
in
terms
of
driving
the
usability
of
these
systems.
It's
only
one.
B
Organ
is,
if
only
one
of
your
engineering
organizations
is
doing
this,
it's
not
as
valuable
to
the
rest
of
the
engineering
org
in
terms
of
the
observability
I'll.
Have
these
missing
pieces
that
prevent
me
from
reasoning
completely
about
the
system
and
these
goals
are
really
congruent
with
the
other
departments
and
in
in
your
in
your
business
and
it's
important
not
to
keep
these
things
a
secret
one
of
the
things
that
was
really
effective.
B
So
all
of
our
micro
services,
hundreds
of
Mike
services
here-
are
the
top
five
slowest
ones
and
here
at
the
top-
and
here
was
the
downtime
and
it
wasn't
about
blaming,
but
it
was
about
you
know,
sharing
that
knowledge
across
the
organization
and
sharing
our
triumphs
as
well.
For
a
long
time,
we
didn't
have
any
downtime
for
any
services
for
several
months,
for
instance,
so
new
approaches
to
consider
in
this
modern
world
AI
machine
learning,
that's
something
that
I
think
is
just
catching
on.
B
There's
a
lot
of
software
services
that
are
built
on
this
primarily
on
this
idea,
but
more
and
more
systems.
If
the
Elcor
be
a
are
starting
to
integrate
things
like
anomaly,
detection
and
so
I
think
you'll
see
more
machine
learning,
even
at
the
basic
sense
catching
on
across
observability,
adopting
a
stream
processing
infrastructure
or
something
I
see
in
larger
organizations
now,
but
I.
Think
more
and
more
people
should
look
at
that.
Instead
of
using
HTTP
api's
chaos,
engineering
is
really
powerful.
B
Now
that
we
have
these
reliable
systems
and
like
kubernetes
cluster
and
all
the
applications
don't
gun
down
for
two
months,
it's
harder
for
me
to
predict
failure,
it's
harder
for
me
to
write
monitors
when
my
system
ever
failure.
They
also
injecting
failure
into
your
system
with
chaos.
Engineering
is
the
new
is
the
new
approach
you
should
be
looking
at.
Since
we
have
these
more
reliable
systems,
testing
and
production,
it's
an
important
thing.
It's
especially
when
you
are
at
a
fairly
large
scale.
B
It's
it's
quite
important
to
look
at
your
production
systems
and
see
if
certain
users
and
certain
geographies
have
problems
and
sampling
sample,
depending
on
your
scale.
Some
things
you
might
want
to
sample.
If
you
don't
have
to
sample-
that's
always
nice,
but
sometimes
it's
been
about
inevitable,
that
you
do
need
to
sample.
So
one
of
the
challenges
we
have
today
briefly
talked
about
this
a
little
bit,
but
percentiles
are
misleading
at
scale.
B
If
I'm
dealing
with
tens
of
millions
billions
of
requests
a
day,
the
99
and
even
the
99.9
percent
I'll
could
be
a
fairly
large
number
of
users,
and
you
don't
necessarily
know
who
those
users
are.
You
should
care
about
every
user,
but
that
one
user
might
be
an
important
business
business
partner
that
was
airing
out
so
being
able
to
know
when
they
said
they
had
our
problem
being
able
to
go,
find
their
requests
and
track
them
down
via
IP
can
be
useful.
You
know
given
GDP
our
concerns
and
things
like
that.
B
Those
that
may
not
always
be
possible
but
being
able
to
track
down
those
individual
request
passes
really
important.
Log
amounts
and
sizes
are
just
going
to
increase
as
people
get
more
comfortable
using
log
aggregation.
One
of
the
challenges
with
building
observability
systems
is
scaling
in
spiky
workloads
and
sometimes
I
may
need
to
over
build
my
system
in
order
to
handle
those
spikes,
but
then
I'm
losing
costs.
So
it's
the
observability
systems
we're
building
today
aren't
quite
as
flexible,
maybe
as
they
could
be.
B
It's
hard
to
run
certain
a
certain
amount
of
these
tools
yourself,
which
makes
it
easier
to
go
to
it
to
assess
so
maintaining
a
large
elasticsearch
cluster
can
be
complex.
For
instance,
today
many
of
the
edge
integrations
we
have
if
I
use
a
fast
layer
cloud
front
the
way
I
get
that
data
into
my
system
is
so
fairly
asynchronous.
B
I'm
generally
reading
off
of
s3
or
they're
batching
up
things
to
send
me
so
I,
don't
necessarily
have
that
real-time
view
of
what's
happening
on
the
edge
a
lot
of
the
time,
we're
shoehorning
into
existing
tools,
and
this
leads
to
a
lot
of
technical
debt.
So,
when
you're,
making
these
choices
about
your
systems,
you're
looking
to
move
into
another
system,
don't
take
those
choices
lightly,
really
consider
the
trade-offs.
I
think
SAS
can
be
a
challenge
because
it
gets
fairly
costly
at
scale.
B
So
when
you
start
out,
you
have
20
micro
services
and
a
few
million
requests
a
day.
It's
it's
all.
It's
all
fairly
reasonable.
When
you
start
getting
into
the
thousands
of
containers
and
lots
of
traffic,
this
SAS
scales
can
go
up
too
high
and
then,
by
the
time
you're.
At
that
point,
it's
really
difficult
to
build
your
own,
so
making
those
choices
or
being
aware
of
those
choices
that
you
make
in
this
trade-offs.
Upfront
is
very
important
and
too
many
engineering,
orgs
I,
see
monitoring
four
different
engineering
groups
in
the
organization.
B
Some
might
use
New
Relic
some
might
use
data
dog
some
use,
elasticsearch
some
use
Splunk
having
one
approach
in
your
organization
is
one
of
the
most
valuable
things
you
can
do
and
putting
all
of
the
data
you
know.
At
least
you
know,
even
if
they
are
in
separate
clusters,
due
to
maybe
security
reasons,
having
the
same
approach
and
not
having
two
different
log.
Aggregation
systems
is
still
the
right
choice
in
my
mind,
so
intuition
is
still
very
powerful,
necessary
tool.
The
software
isn't
solving
that
for
us
today.
B
It's
still
important
to
have
trained
operators
who
understand
distributed
systems
to
understand
Software,
Defined,
Networking
routes,
firewalls,
all
those
things
are
important
to
have
the
observability
tools
just
really
infer
our
intuition.
It's
never
going
to
replace
our
intuition
uptime
and
mean
time
to
recovery
are
important
still.
So
the
fact
that
my
website
isn't
is
up
or
down
is
still
important.
I,
don't
want
to
say
it's
not,
but
now
that
we
have
these
fairly
reliable
systems,
even
in
a
multi
region,
multi
cloud
that
Ferren
that
never
go
down.
B
That's
when
we
really
need
to
our
next
focus
needs
to
be
the
quality
user
experience
and
really
caring
about
that,
because
we
already
have
these
reliable
systems.
You
know
five
nines
and
things
like
that.
Easy
uptime
is
far
easier
to
come
by
it's
far
easier
to
have
a
load
balancer
and
a
cloud
organization
that
can
scale
to
tens
of
terabytes
a
day
of
ingest,
so
take
advantage
of
that
and
focus
in
on
the
user
experience
in
your
system.
So
some
of
the
best
practices
you
should
look
at
adopting.
B
Remember
that
not
one-size-fits-all
are
doing
many,
be
mindful
of
the
trade-offs
that
you're
making
when
you
choose
to
use
these
systems
go
beyond
your
your
engineering
organization.
Perhaps
you
should
be
pulling
in
data
from
your
CRM
about
you
know,
signups
or
sales
or
your
Zendesk
tickets
lining
up
the
amount
of
Zendesk
tickets.
You
have
during
a
certain
time,
can
help
you
correlate
information
about
users
and
their
geography
in
your
observability
system
being
sure
to
balance
the
cost
and
usage
trade-offs.
B
One
of
the
things
that
I've
when
we
implemented
distributed
tracing
that
Bebo
was
that
people
rarely
used
it.
We
were
spending
a
good
amount
of
money
to
store
this
data
in
elasticsearch
cluster
and
maybe
three
user.
You
three
users
used
it
a
month,
and
so
we
were
storing
thirty
days
of
data
and
we
made
the
choice
to
only
store
14
days,
14
days
of
data.
Because
of
that,
so
you
should
really.
You
know
be
aware
of
these
trade-offs.
B
In
terms
of
you
know
the
cost
and
usage
and
make
sure
people
are
actually
utilizing
the
tools,
if
they're
not
utilizing
the
tools,
understand.
Why
and
see
you
know
see
if
there's
an
alternative
approach
and
I
mentioned
this
earlier,
but
have
your
observability
config
live
with
your
applications,
deploy
it
with
your
CD
system
and
bonus
points.
B
So
if
you
are
using
distributed
tracing,
consider
putting
generating
your
correlation
IDs
out
on
your
client
from
your
web
application,
that's
I
think
a
really
powerful
approach
that
people
need
to
be
start
start
using
be
wary
of
education
because
you
can
lose
contacts,
especially
in
a
global
environment.
So
bringing
you
know
putting
together
metrics
you're,
just
depending
on
your
your
city
ends.
Metrics.
B
You
end
up
building
a
much
better
observability
system,
so
going
a
little
bit
slower
than
then
I
thought
so,
hopefully,
we'll
stop
time
for
questions.
I
should
be
able
to
wrap
this
up
fairly.
Quick,
I'm,
just
gonna
briefly
talk
about
the
future
and
what
I
see
coming
and
observability
we're
all
going
to
have
cheaper,
better
faster
systems
and
clouds.
That's
you
know,
that's
not
going
to
change
I!
Think
more
and
more
workloads
are
going
to
be
moving
to
the
cloud.
There's
lots
of
talk
these
days
about
you
know,
hybrid
cloud
solutions.
B
Aws
has
one
Google
has
one!
So
that's
going
to
continue.
The
tools
were
running
are
still
fairly
complex.
At
times
and
hard
to
manage,
that's
going
to
get
easier
and
there
I
think
in
general,
there's
going
to
be
less
tools
in
terms
of
there's
a
lot
of
tools
that
do
the
exact
the
same
thing
and
in
general
orgs
are
using
several
different
tools
for
APM.
B
A
native
part
of
the
infrastructure
will
be
something
that
big
shops
only
is
Kafka,
and
things
like
that
you'll
see
more
and
more
people
rely
on
streaming
to
move
data
around
between
their
ap
is
or
or
their
they're
logging
systems
and
doing
less
HTTP
I
calls
a
less
parsing
of
files
and
shipping
places
we're
already
seeing
observability
engineers
and
orcas
today,
I
see
job
postings
for
people
specifically
focused
on
in
an
observability
similar
to
what
Sree
was
becoming
several
years
ago.
So
it's
really
becoming
a
discipline
which
I
think
is
a
really
interesting
thing.
B
So
as
we
get
all
this
data
more
and
more
data
having
the
ability
to
look
at
a
subsection
of
that
data,
so
flipping
a
bit
in
my
system
like
a
feature
flag
in
order
to
turn
the
clients
in
Brazil
on
to
debug
a
problem.
So
you
know
telling
the
clients
in
Brazil
to
record
their
DNS
entries
or
their
time
to
first
frame
in
the
case
of
vivo
is
going
to
be
something
we're
actually
going
to
see
is
actually
having
these
dynamic
observability
systems
all
the
way
out
to
the
client.
B
Your
repos
I
think
a
lot
of
these
things
will
be
more
and
more
commonplace
over
the
next
10
years.
So
how
do
you
go
about
building
your
observability
culture?
It's
never
over.
You
always
need
to
be
asking
new
questions
as
you
release
new
products,
releasing
the
features,
the
building
questions
about
those
systems
and
towards
observability
system.
It
always
needs
to
be
evolving
champing.
B
The
business
to
inform
you
on
what
your
focus
should
be
around
users
in
terms
of
the
amount
of
trial
like
setups
or
trial
Creations,
you
have
verses
permission
users
and
judging
the
happiness
of
the
quality
of
the
trial
users
versus
of
production
users.
So
the
understanding
what
your
business
is
about
and
what's
important,
your
business
will
help
inform
you
as
to
what
you
should
conserve
so
false,
fair
culture,
culture
of
accountability
within
your
organization,
so
you
and
your
up
times
and
errors
just
because
AWS
was
down.
That
was
the
decision.
B
Your
organization
made
to
go
in
you
know,
maybe
only
in
one
region
or
not.
You
know
not
take
on
that
complexity,
so
own,
owning
owning
your
decisions
and
owning
your
observability
is
really
important
to
building
these
systems
and
making
them
reliable,
empower
all
of
your
engineers
to
make
it
easy
to
use
these
systems
everyone.
You
know,
instrumenting
the
code
carrying
about
logging,
how
they're
logging
the
number
of
tags
they're
using,
but
it
needs
to
be
easier
for
your
engineers
to
do
this
rather
than
you
know,
a
dictate
and
pull
request.
B
And
finally,
the
most
important
thing
is
having
a
culture
of
empathy
for
you,
the
users
and
each
other,
we're
all
working
in
businesses
towards
these
common
goals
and
it's
usually
about
our
product
or
about
the
people
using
our
service.
So
having
empathy
for
your
users
and
the
quality
of
their
their
experience,
but
also
that
person
that
got
paged
because
you
know
I
wrote
a
bad
line
of
code
or
I
didn't
have
a
proper
timeout.
It's
important
to
remember
that
we're
all
in
this
together,
so
I'll
close
with
a
quote
from
Brian
Knox
from
digitalocean.
B
The
goal
of
observability
team
is
not
to
collect
logs
metrics
or
traces
is,
is
a
build,
a
culture
of
engineering
based
on
facts
and
feedback
and
then
spread
that
culture
within
the
organization.
So
just
a
few
people,
I'd
like
to
thank--
at
the
end,
I
don't
know
how
much
time
we'll
have
for
questions.
B
B
Maybe
you
were
both
54in
combined,
see
me
all
actually
being
barcelona
for
that,
and
if
you
want
to
know
more
about
gmail,
you
can
try
it
for
free
for
30
days,
and
we
also
have
a
free
cloud
chair
that
you
can
use
as
well
for
I
think
less
than
a
gigabyte
a
day
of
traffic.
So
thank
you
very
much
and
I
will
try
to
bring
up
questions.
B
Are
there
any
questions
up
there?
It
is
what's
kind
of
carrots.
Do
you
give
engineers
to
encourage
encourages
of
the
observability
practice?
So
a
lot
of
these
carrots
that
I
give
or
the
way
that
I
think
about
it
is
to
make
things
easy.
So,
if
I
make
things
easy,
so
if
I
define
that
every
application
in
my
organization
has
a
CTO
or
as
a
VP,
every
application
has
to
track
the
number
of
500s.
B
So
if
I,
if
my
sorry's
create
a
library
to
make
that
happen,
it's
easy
for
the
engineers
to
to
integrate
those
metrics
into
into
their
systems.
So
I
think
it's
a
combination
of
making
it
easy
for
your
engineers,
but
also
if
they're,
getting
paged
it'll
be
easier
for
them
to
debug
their
systems
and
other
people's
systems
and
will
make
their
on-call
much
much
better.
B
So
I
think
these
are
the
carrots
that
you
bring
out
and
the
last
question
I'll
answer:
there
were
two
I'm
gonna
explain
how
chaos
engineering
later
observability
I
eluded
it
to
it
a
little
bit
but
more
explicitly
when
we
have
these
fairly
reliable
systems
with
no
downtime
I
can
keep.
You
know
I
didn't
write,
kubernetes
cluster
didn't
go
down
for
you
know
two
years
and
you
had
uptime
for
months
across
all
of
our
applications.
B
It's
hard
to
know
how
your
system
is
going
to
react
when
you
don't
know
how
it
breaks
so
by
using
a
chaos,
engineering
product
like
chaos,
IQ
or
chaos,
monkey
and
injecting
these
failures
into
their
systems.
I
can
see
what
happens
when
I
take
down
a
micro
service
or
I,
take
down
a
region
and
was
on
availability
zone
and
I
can
see
how
my
systems
react
and
then
I
can
know
what
monitors
I
need
to
set
up.
So
I
can
react
to
those
problems.
So
I
think
it's
important
to
start.
Looking
at
that.
B
In
addition,
you
can
make
it
part
of
the
or
CIA
CD
set
up.
So
when
you're
doing
your
your
your
CI,
you
can
actually
inject
failures,
be
it
with
load
balancers
or
any
part
of
that
infrastructure
to
actually
learn
more
about
that.
So,
if
you
want
to
learn
more
about
that,
we
actually
have
an
e-book
that
that
we
released
on
on
that.