►
From YouTube: Sponsored Talk: Planning for Observability
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2021 Virtual from May 4–7, 2021. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Sponsored Talk: Planning for Observability
Open Telemetry opens up a path for the fully open-sourced instrumentation of critical applications, but whether your strategy involves self-hosted open source or vendor tools, many of the challenges remain the same. In this short talk, Nica will cover the key strategies for ensuring observability. We’ll cover:
- What is observability anyway?
- Towards observable architecture
- Structured logging
- Usability of monitoring data
- Growing your team’s observability skills."
A
This
is
my
talk
planning
for
observability,
where
we're
going
to
talk
about
some
of
the
strategies,
key
decision
points
and
sort
of
pre-planning
that
can
help
you
create
a
more
observable
application.
We've
only
got
a
20
minutes
for
this
talk,
so
I'm
just
going
to
go
ahead
and
jump
right
in
pretty
quick
here.
A
So
my
name
is
nochnika.
Nika
is
fine.
If
you're
having
a
little
trouble
pronouncing
nica,
I
am
a
developer
advocate
for
new
relic.
I
specialize
in
cloud
architectures,
especially
serverless,
architectures
and
other
kinds
of
cloud
specific
stacks.
I
do
do
other
observability
work.
I've
been
working
with
new
relic
and
observability
for
a
number
of
years.
I
tweet
at
serverlessmom.
A
So
there
then
my
serverless
serverless
pedigree
does
show
up,
I'm
also
on
twitch,
where
I
mainly
twitch
a
a
stream
cloud
engineering,
stuff
building
applications,
occasionally
making
other
fun
stuff
coding,
drum
machines
and
other
cool
goofiness,
but
yeah.
That's
that's
a
very
dev
centric
stream.
If
you
want
to
come
check
it
out,
so
since
we
only
do
have
20
minutes,
I
will
dive
right
into
it
and
the
first
one
is
this
thing
that
I
talk
about
quite
often,
which
is
what
is
observability
exactly
so
on
the
west
coast
of
the
us.
A
Observability
is
a
really
big
buzzword.
A
lot
of
people
feel
like
it's
just
some
marketing
term.
It
just
means
you
know,
you
know
it's
just
like.
Instead
of
calling
it
a
task,
you
call
it
an
action
item
and
that
just
sort
of
sounds
cooler
than
task.
You
know
maybe
observability
just
sounds
cooler
than
logging
or
maybe
observability
sounds
cooler
than
metrics,
and
you
know
really.
Observability
is
a
collective
term
that
that
means
a
number
of
things
and
it
really
covers
not
a
particular
technology
but
sort
of
a
design
goal.
A
I
mean
that
technically
right,
that's
a
problem
with
very
poor
observability
but
relatively
easy
to
solve,
but
obviously
that's
not
great.
That's
not
how
we
want
to
live
our
lives.
We
want
to
look
at
a
dashboard,
a
log
or
portable
problem.
We
want
to
be
able
to
very
quickly
go.
I
know
what
I
need
to
do
next
to
solve
that.
A
So
when
that
time
is
very
low,
we
have
high
observability
when
that
time,
when
it's
taking
us
a
long
time
to
get
to
understanding,
we
have
particularly
low
observability,
and
this
is
a
quote
from
a
serverless
developer.
This
is
patrick
zigger.
It
really
stuck
with
me
it's
a
bit
of
a
worthy
slide,
but
I'm
going
to
read
it
out
right.
He
said
I
can
scale
serverless
as
big
as
you
like,
a
random
500
error
that
I
can't
diagnose.
A
Is
that
before
they
face
problems
with
the
complexity
of
the
stack
and
way
before
they
face
big
cost
issues,
observability
call
it
you
know.
Sometimes,
specifically,
you
know
the
debugging
of
new
stacks.
It
comes
up
as
a
much
bigger
problem
way
earlier
in
the
process,
and
you
know
that's
kind
of
the
nature
of
the
beast,
maybe
because
some
of
the
scaling
problems
that
we
dealt
with
previously
or
some
of
the
config
issues
that
we
would
deal
with
previously
are
now
much
more
automated.
A
A
So
one
of
the
things
about
defining
this
problem,
this
way
is
that
other
stuff
comes
up
as
having
an
effect
on
observability
right.
The
speed
with
which
a
dashboard
can
be
read
has
a
direct
effect
on
observability,
and
that
is
a
really
key
observation.
This
is
right
up.
Nice,
prometheus,
grafana
dashboard.
A
I
believe
this
is
actually
using
open,
telemetry
instrumentation,
to
generate
this.
One
though
like
when
I
grab
the
screenshot,
but
you
know
take
a
look
at
it.
Even
in
this
tiny
version.
I
think
that
you
can
see
right
that
a
pattern
is
readily
observable
and
that's
really
a
ui
issue.
That's
a
graphical
issue.
It's
it's
not
about
what
data
is
underlying
that
dashboard.
Obviously,
we
need
the
data
to
be
reliable,
but
the
production
of
more
data
isn't
necessarily
helpful.
A
In
fact,
it
would
hurt
observability
because
we
go
to
look
at
the
logs
we'd
have
all
this
junk
and
all
this
noise
that
would
be
hiding
whatever
useful
information
might
be
in
there
I
mean-
maybe
nothing
useful
is
in
there
right
now,
but
even
so,
we
would
still
have
worse
observability
than
when
we
started,
because
we
would
need
to
spend
so
much
time
digging
through
every
time.
We
wanted
to
get
some
information,
so,
let's
think
a
little
bit.
A
This
is
just
some
very
simple
information.
We
want
to
see
error
rates
on
each
component
right
because
we
want
to
know
hey
is
the
is
the
gateway
throwing
a
bunch
of
errors.
Is
the?
Is
the
web
application
layer
throwing
a
bunch
of
errors?
Are
we
seeing
them
in
the
database?
Where
are
we
seeing
some
errors?
Even
being
reported
we'd
like
to
see
throughput,
I
say
on
the
back
end
here
I
mean
sort
of
on
the
on
the
rear,
most
component
right,
the
component,
that's
the
the
furthest
to
the
right
on
our
little
little
graph.
A
So
if
we
see
hey
we're
having
some
problems-
and
we
see
especially
that
the
throughput
on
like
the
database
has
just
dropped
off
a
cliff,
then
we
start
to
see
okay.
Well,
I
have
a
pretty
good
theory
of
at
least
what
the
nature
of
the
problem
is
right
and
looking
into
like
the
configuration
of
the
database.
A
Server
is
probably
not
gonna,
be
a
waste
of
our
time,
because
the
source,
the
problem
is
almost
certainly
right
upstream
from
there
then
it'd
be
really
nice
to
look
at
changes
to
our
permissions
and
configuration
and
we're
going
to
talk
about
infrastructures
code
and
some
of
the
other
steps
that
can
really
improve
observability
here,
because
those
changes
really
can
you
know
often
for
a
problem
again.
You
know
my
my
sort
of
comfort
zone
is
often
in
these
cloud
architectures.
So
this
is
a
pretty
frequent
problem
right.
A
The
configuration
to
what's
allowed
to
talk
to
what,
obviously
we
need
those
configs
to
be
available,
but
when
they
get
messed
up
right,
then
we
have
our
stack
suddenly
failing.
The
last
is
deployment
history
right
and
and
totally
before.
We
have
any
kind
of
deep
observation
tools.
These
are
all
things
where,
if
we
have
them,
our
observability
is
a
lot
higher
for
that
random
500
error
from
our
endpoint.
A
So,
let's
talk
about
picking
the
right
tools
now,
of
course
you
know
it's
open,
telemetry
community.
Today
we're
not
going
to
be
talking
about
picking
the
right
measurement
tools,
but
the
other
tools
around
that
that
can
help
us
get
information
or
can
be
just
defined
strategically
or
defined.
Tactically
for
our
team,
so
infrastructure's
code.
A
So
you
know
I'll
I'll,
see,
questions
like
well,
you
know:
can
we
use
terraform
right?
Can
we
use
staggering?
Oh,
it's
nice
to
use
terraform
because
multi-cloud,
maybe
that's
the
better
solution
right.
Should
we
use
serverless
framework
to
deploy
our
serverless
environment
should
we
use
some
kind
of
vendor-specific
tool
like
the
cloud
deployment
kit
from
aws,
and
you
know
the
real
thing
that
I
see
as
a
theme
is
not
so
I
don't
see
a
lot
of
people
regretting
that
they
picked
a
particular
infrastructure
as
code
tool
for
deploying
changes.
A
The
infrastructure
I
see,
people
glad
that
they
did
it
as
opposed
to
not
doing
it.
So
if
you're,
storing
a
bunch
of
config
file
in
a
readme
somewhere
that
is
sort
of
getting
bounced
around
your
team
and
you're
having
to
write
in
that
readme
file,
maybe
you're
going
back
and
bolding
stuff
to
be
like
hey,
be
sure
that
you
make
this
setting
when
you
set
up
a
new
instance
right.
A
That
means
we're
sort
of
drifted
away
from
infrastructure
code
and
we're
probably
gonna
face
problems
when
you
know
you're
not
available
or
when
there's
team
changes,
and
you
see
problems
with
this
infrastructure.
So
infrastructure's
code
is
a
key
key
piece
of
observability
because
it
lets
us
go
and
get
clues
about.
What's
actually
going
on
and
what's
changed
recently
and
try
to
make
those
correlations
in
time.
A
So
what
we
get
from
infrastructure's
code
is
we
get
a
commit
history
right?
We
get
clear
communication
of
the
team
of
you
know:
hey
what's
changed
and
by
whom-
and
this
is
this-
isn't
about
letting
blame
at
all.
This
is
about
knowing
who
to
talk
to
or
knowing
what
the
possible
motivations
for
changes
might
have
been,
and
then
the
last
one
is
well-documented
requirements
and
you'll
be
surprised.
How
often
this
comes
up
where,
by
documenting,
for
example,
that
the
memory
needs
of
certain
instances
has
changed.
A
A
So,
let's
talk
about
structured
logging.
The
the
diagram
up
here
is
some
my
own,
my
own
attempt
to
render
some
penrose
tiles
and
I've
made
actually
a
couple
mistakes,
but
I
have
it
in
here
because
it's
one
of
the
most
beautiful
examples
of
structure
with
variation
since
pemero
styles
sort
of
appear
to
be
periodic,
but
have
slight
changes
that,
don't
repeat
so.
A
You
know
it
has
that
structure
and
I,
like
that,
a
lot
and
also
roger
penrose
just
won
the
nobel
prize
for
physics
just
a
couple
weeks
ago,
so
I
was
throwing
a
little
plug,
but
so
totally
outside
again.
This
is
not
a
tool-specific
recommendation.
I'm
not
saying
use
this
one.
Logging
library,
of
course
that's
going
to
vary
by
what
language
you're
in
for
starters,
but
just
the
concept
of
structured
logging
is
such
such
a
key
piece
of
getting
real
information
about.
What's
actually
going
on
inside
our
stack.
So
oh
slides
around
order.
A
Sorry
about
that,
so
I
think
you
know.
Structured
logging
is
something
that
you
can
just
do
completely
on
your
own,
where
you
just
say
hey.
Instead
of
just
having
a
simple
line
of
text,
I'm
going
to
have
an
object
here
with
a
few
key
value
pairs
in
it,
and
maybe
you
add,
a
key
value
pair
as
needed.
Maybe
you
update
the
ones
you
already
have.
A
It
just
makes
it
that
much
easier
to
parse
through
this
history
later
and
make
your
interactive
search
tools
and
maybe
even
make
a
nice
visual
dashboard
based
on
what's
being
put
in
your
logs.
So
I
also
really
want
to
mention
that
a
key
key
piece,
open
telemetry
specific,
is
an
ability
to
contextualize.
Your
logs
is
so
so
great.
A
Being
able
to
contextualize
logs
with
metrics
with
traces
is
a
a
key
performance
feature,
something
we
pursue
at
new
relic
as
well,
because,
along
with
having
some
kind
of
structure
to
allow
you
to
query
in
being
able
to
connect
a
trace
to
a
log,
is
an
absolute
game
changer
for
your
ability
to
actually
get
some
insight,
and
that
especially
comes
up.
You
know.
Obviously,
these
problems
are
usually
obvious
when
they
happen
as
part
of
an
incident
and
an
outage
and
everybody's
sort
of
flipping
out
and
we're
all.
A
A
So
I
you
know
it
is
the
first
warning
we
have
that
something
that's
gone
wrong,
and
maybe
you
know
we
start
looking
into
something
not
based
on
learning
based
on
you
know,
complaints
of
the
customer
or
reports
internally,
but
very
often
the
nature
of
alerts
is
a
powerful
way
for
us
to
have
a
first
clue
about
what
we're
looking
for
when
we
go
and
dive
into
our
logs
or
dashboards.
Or
what
have
you
so
one
key
point
I
want
to
mention
is
think
about
low
and
high
alert
threshold.
A
So
one
of
the
examples
from
my
500
error
scenario
was
hey:
let's
look
at
throughput
on
the
database
and
so
that's
a
really
good
use
of
that
low
thresholding
to
say,
hey.
We
have
a
problem
because
we
have
no
traffic
through
here
and
my
little
quote
around
this
is
right.
Ten,
a
ten
time.
Growth
and
page
view
is
a
concern
as
as
is
zero
time
growth
in
page
views
right.
A
If
we
suddenly
see
that
a
section
of
our
site
or
a
particular
set
of
api
routes
has
stopped
being
used
at
all,
it's
worth
throwing
an
alert
there.
Now
you
know
maybe
that
that's
that's
predictable,
for
race
reasons,
it's
predictable
because
of
low
traffic,
and
so
it
is
important
to
check
in
and
make
sure
your
alerts
make
sense,
but
especially
because
open
inflammatory
opens
up
the
route
to
get
a
ton
more
metrics
and
a
ton
more
stuff
to
alert
off
of
thinking
about
low
thresholds
is
a
really
key
piece.
A
So
let's
talk
about
creating
observability
with
the
the
abilities
of
your
team.
So
this
is
an
old
joke,
but
it's
it's
still
applicable
right.
You
know
we
need
a
devops
ninja.
You
know
they
need
five
years
of
service
experience,
10
years
of
kubernetes
experience,
I
got
to
move
to
greenland
to
do
the
job.
Well,
okay,
I
would
move
to
greenland
at
the
drop
of
a
hat.
I
would
I
would
I
would
oh
boy
that
sounds
cool,
but
beyond
that.
Of
course,
this
is
a
you
know.
It's
an
ad.
A
It
is
tough,
so
one
of
the
takeaways
I'd
love
you
to
get
from
this
is
that
when
you
have
a
great
team
that
has
great
observability,
you
really
is
something
that
you
grow
and
it's
not
something
that
you
go
out
and
hire,
and
I
mean
you
know.
Why
am
I
talking
about
the
people
side
of
this
at
all?
I
would
say
you
know
remember.
A
So
the
three
pieces-
and
this
is
not
my
comparison-
this
is
sheen
brussels
lego,
who
first
proposed
this.
I
really
liked.
It
is
that
you
find
the
good
seeds
you
find
people
either,
who
are
new
and
very
eager
to
you,
know,
fully
understand
their
architecture
become
sort
of
as
full
stack
as
possible,
or
you
find
people
who
really
you
know
they
have
a
ton
of
experience,
but
they
love
applying
it
in
new
environments,
and
you
want
to
give
that
team
the
soil
that
it
needs
to
really
develop
its
skills
in
a
complete
way.
A
This
example
started
with
right
what
happens
when
we
lose
some
senior
engineers,
and
you
know
more
than
anything
when
I
talk
to
people
who
are,
you
know,
have
real
experience
on
why
they
leave
a
lot
of
things
they
talk
about.
Is
they
say?
Well,
I
wasn't
growing
there.
You
know
my
knowledge
wasn't
growing.
I
was
doing
stuff
I've
done
before,
and
so,
if
you
give
them
those
chances
to
grow,
that's
really
gonna
build
your
ability
to
retain
good
people.