►
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
On the OSS Path to Full Observability with Grafana - David Kaltschmidt, Grafana Labs
Grafana is coming “off the wall”. To make it more useful for interactive debugging, David and his team have already integrated two pillars of observability - metrics and logs. They are currently adding tracing to complete the incident response experience. All to minimise the cost of context switching during those crucial minutes after getting paged. This talk will demonstrate the various methods we’ve used to link the data together. Prometheus is providing the metrics. Via its histograms, request latencies can be extracted to inform each tracing span from Jaeger. Grafana also ensures that lines from your log aggregation system are annotated with span and trace IDs, as well as the other way around: associating logged values with spans. David will show how these OSS parts should be deployed to achieve full observability in an engaging user experience that saves valuable minutes.
To Learn More: https://sched.co/GrXC
A
I
think
it's
summer
get
started
alright,
so
people
coming
in,
but
we
just
go
ahead.
All
right.
Welcome
everyone.
I
think
this
is
the
first
session.
Let's
see
how
this
goes,
you
are
all
with
me
on
the
path
now
to
flow
observability
with
open
source
software
and
I
am
David
I'm,
director
of
user
experience
at
Cortana
labs,
which
basically
means
you
can
either.
A
If
you
get
stuck
somewhere,
you
can
either
shake
your
fist
or
you
can
write
to
me
and
then
maybe
I
can
fix
the
button
where
you
get
really
good
start
and
just
give
you
a
quick
outline
of
the
talk.
Gonna
give
a
quick
intro
of
khokana,
then
we're
gonna
make
an
app
fully
observable
and
then
we're
gonna
look
at
logging
and
psych
profile
in
a
bit
more
detail.
So
first
quick
intro,
a
quick
question
who
here
is
using
grow,
follow
that's
a
fair
amount,
I'm.
A
A
A
The
idea
is
that
then,
at
this
point
you
will
consult
your
logs
through
some
log
aggregation
system,
and
sometimes
you
might
also
need
a
distributed
tracing
to
the
back
latency
issues,
for
example,
and
the
general
idea
is
that
once
you've
sort
of
gone
through
all
these
motions,
that
you
have
a
vague
idea
on
what's
going
wrong,
and
that
will
also
give
you
an
idea
on
how
to
fix
it.
So
yeah
graph
Anna
wants
to
help
you
enable
this
whole
journey.
So
what
does
that?
Look
like
we're
gonna
instrument,
an
example
app.
A
Let's
look
at
a
classic
3t
app.
You
will
have
a
local
answer
at
the
top.
An
absolute
bunch
of
app
servers
in
the
middle
and
a
free
database
servers
that
that
return,
the
data
to
the
app
servers
and,
let's
assume
it's
it's
deployed
to
kubernetes
and
it's
running
fine,
although
do
we
know
it's
running
fine
I
mean
who
knows
right.
A
The
problem:
is
it's
not
really
instrumented
yet
so
we're
gonna,
we're
gonna,
add
this
and
the
first
or
the
most
basic
form
of
instrumentation,
it's
usually
logging
and
because
logging
is
quite
or
it's
quite
important,
you
can
think
of
what
a
lock
line
represents.
It's
basically
where
the
code
path
executed
at
this
particular
point
in
time,
and
if
you
have
enough
of
those,
you
could
theoretically
piece
together
what
your
code
was
doing
at
which
point
in
time.
A
So
the
idea
was
also
that
logging
is
better
than
not
logging
at
all
right
and
then
you
can
also
do
the
next
step
that
a
structured
logging
could
be
better
than
just
logging
and
in
this
example,
we're
with
the
golang
app
that
we're
in
store
where
we're
using
a
lock
for
matter
that
comes
with
with
with
go
kit,
and
this
will
give
us
nice
key
value.
Log
lines
I'm
going
to
show
later
how
that's
helpful
so
onto
metrics
what
are
metrics
or
why?
Are
they
helpful?
A
So,
basically,
if
you
looked
at
the
log
lines,
you
will
have
a
you
will
have
a
difficult
time
to
to
piece
together.
For
example,
a
request
rate
or
an
error
rate
I
mean
you
can
sort
of
do
the
math,
but
it'll
be
much
easier,
and
if
something
else
did
this
for
you
and
there
are
established
metric
systems
out
there
and
Publius
is
one
of
those
and
here's
an
example
on
how
we
can
extend
the
same.
The
same.
Go
line
app
with
a
function
wrapper
around
the
HTTP
handler
that
will
give
us
the
request.
A
A
There's
I
mean
it's
not
it's
not
completely
trivial
to
hook
this
all
up
together.
So
this
is
a
really
good.
Is
a
really
good
set
of
resources
to
consult
on
how
to
get
all
these
pieces
working
together?
Because
some
are
you
needs
you
need
some.
You
need
a
you
need,
a
bunch
of
exporters
for
Prometheus
that
will
give
you
the
right
the
right
metrics
out
of
kubernetes,
for
example.
A
Then
then
we
also
publish
our
conflicts
for
running
Prometheus
and
alert
manager
and
go
fauna
together
and
have
all
the
paths
mix
and
match
up
nicely
and
then
the
the
last
link
is
project
where
we're
trying
to
encourage
people
to
publish
their
own,
what
we
call
mix-ins
that
that
should
basically
describe
how
do
how
are
their
own
apps
will
or
should
be
monitored.
So,
for
example,
there's
a
constable
mixin
or
there's
a
kubernetes.
B
A
A
Need
this
back
and
before
I
introduce
the
error,
everything
looked
sort
of
green.
Let
me
just
zoom
in
here
a
little
bit
and
this
dashboard
follows
the
rap
method.
By
alone,
I
want
a
blue
key.
We
have
the
apps
that
come
or
where
the
requests
come
in
from
the
top.
We
also
have
those
apps
at
the
top
and
then,
as
they
go
lower
down
the
stack,
those
rows
should
also
be
lower.
A
So
the
request
comes
in
through
a
local
answer
and
then
it
hits
the
app
and
then
hits
a
database
and
in
the
happy
path,
everything's
green
right,
the
latency,
Zurn
Ison,
my
single
two
milliseconds
I-
think
that's
pretty
good
and,
most
importantly,
there's
no
Aris
right.
So,
but
I
did
so.
You
can
also
tell
here
this
was
before
and
did
I
introduce
the
error
right.
So
now,
let's
assume
someone
deployed
something
that
was
a
bit
that
had
an
issue
and.
A
Still
need
further,
and
now
we
can
also
see
we
can
already
see
two
things
happening.
First
of
all,
the
latency
shoots
up
right
before
it
was.
It
was
all
around
ten
milliseconds,
and
now
we
have
three
milliseconds
in
99th
percentile,
that's
just
be
long.
It's
quite
that's
quite
a
jump.
So
that's
one
issue
or
could
also
be
the
same
issue.
We
don't
know
yet
and
then
here
there's
some
Reds
right,
Red's,
not
really
good.
So
just
probably
errors
yeah,
we
can
really
could
already
see
because
it
is
nicely
formatted
here.
Those
are
five
hundredths.
A
Okay,
so
somehow
there
are
error
errors
being
produced
by
the
database,
but
if
we
scroll
up
again,
luckily
they
don't
make
it
to
the
app
or
it's
real
open
answer.
So
the
the
user
isn't
very
likely
seeing
this
as
errors,
but
the
latency
is
pretty
high
and
this
will
affect
the
the
impact
of
how
our
your
service
is
being
perceived.
And
let's
look
at
the
latency
issue
first
and
for
this
I
like
to
use
Jaeger.
So
let
me
just
quickly
go
there
and
let's
just
run,
let's
just
quickly
narrow
down
on
what
we're
looking
for.
A
If
you're,
if
you
remember
the
latencies,
they
were
around,
they
jumped
up
to
800
milliseconds.
So,
let's
just
zoom
in
on
this
something
between
500
and
1.5
seconds
and
all
right.
Let's
just
look
at
this
example
here
top
here
all
right,
so
this
is
now.
This
is
now
a
trace
from
our
that
that
is
producing
a
really
high
latency
and
we
can.
We
can
step
in
all
through
Obachan.
A
This
is
already
where
I'm,
using
my
brain
from
basically
from
inside
knowledge
that
I
have
about
the
application
right
and
and
if
I
look
at
the
logs,
this
just
confirms
it
right.
There's
a
backing
off
period
with
which
the
the
app
which
the
app
took
before
was
issuing
a
second
request,
which
is
this
one
over
here,
and
this
woman
succeeded
so
to
know.
The
fix
for
this
particular
issue
were
probably
that
the
500
milliseconds
is
a
long
is
a
long
time
like
and
the
whole.
A
The
whole
app
could
have
responded
a
little
quicker
if
the
back
of
period
wasn't
there
high.
So
this
would.
This
would
probably
be
my
fix
for
this
particular
issue,
but
there
was
a
second
one
right.
There
was
the
second
issue
where,
where
they
were
whatever
arrows
right,
so
these
basically
this
read
the
direct
section
up
here
and
now:
I
can
I
need
to
figure
out
which
parts
of
the
database
or
I
mean
I,
don't
know
yet
right,
so
I
would
jump
to
a
new
section
of
Cortana,
which
is
called
explore
this.
A
But
sadly,
someone
was
using
the
alternate
some
by
I,
don't
know
aggregation,
so
I'm
usually
don't
really
know
how
to
operate
with
this.
But
luckily
this
system
should
suggest
the
the
correct
the
correct
labels
they
can
use.
Okay,
so
now
I'm
going
to
now,
I'm
gonna
also
break
this
down,
not
just
by
status
code,
but
also
by
instance,
and
the
just
to
look
just
looking
at
the
table.
A
I
can
already
see
only
one
database
instance:
it's
producing
five,
hundreds,
okay,
so
it
was
good
because
now
I
can
just
click
in
filter
also
by
this
one
and
then
just
to
verify
okay.
Well,
this
clearly
produces
non
zero,
an
error
rate
for
non
zero,
so
I
think
I've
zoomed
in
on
the
correct
the
correct
instance,
and
now
what
we're
really
good
to
just
look
at
the
logs
of
this
particular
instance.
So-
and
I
can
do
this
now
inside
of
katana.
If
I
switch
over
to
griffons
logging
and.
A
A
So,
there's
something
ok,
and
here
we
here
we
can
see
already
the
sort
of
errors
that
were
being
produced
and
what
we're
also
doing
is
in
in
the
browser
we're
pausing
the
log
lines.
So
we
can,
we
can
also
see
not
just
what
sort
of
errors
were
being
thrown,
but
also
the
the
distribution
here
so
and
I
can
see.
Ok,
there's
a
lot
of
Korea
logged
timeouts,
but
also
too
many
open
connections,
and
this
is
nowhere.
A
I
need
to
use
my
brain
again
to
with
inside
knowledge
of
the
application
and
I
know
that
probably
the
too
many
open
connections
are
causing
the
query,
lock,
acquisition
era,
and
but
this
still
helped
me
to
figure
out.
Ok,
now,
probably
at
the
root
cause
with
with
the
too
many
connections
and
the
solution
for
this
would
very
likely
be
to
have
more
replicas
of
the
database,
for
example,
or
I
could
use
connection
pooling
in
the
app
client
when
it
connects
to
the
database.
A
Ok,
so,
and
with
these
two
fixes
now
in
mind,
I
can
I
can
I
can
deploy,
fix
and
I'm
at
the
end
of
my
journey
so
and
just
to
showcase
their
logging
a
bit
more.
There's
not
just
the
data,
if
they're,
not
just
logging
from
this
database
component,
but
also
everything
else,
that's
profound
a
spec
and
is
running
its,
for
example,
because
this
data
source
is
pointing
to
the
finest
internal
system
and
we're
running
Cassandra,
for
example,
and
I'm
using
this
label
select
a
language
that
is
similar
to
Prometheus.
A
And
if
you,
if
you
remember
what
I
did
before
I
was
zoning
in
on
the
on
the
database
and
a
particular
instance
during
you
are
using
propio
when
I
was
looking
at
the
metrics
and
then
I
was
switching
over
to
the
logging
system
and
I
was
using
the
same
at
the
same
labels
yeah.
And
if
you
think
that,
if
you
think
of
what
the
log
line
represents
is
not
just
the
string
but
a
graph,
honest
logging
also
attaches
or
some
metadata
fields
that
we're
indexing.
A
A
Here
here
no,
we
have
to
the
lock
format,
key
value
format,
and
here
we
can
we
can.
We
can
definitely
pass
all
these
all
these
keys
for
you
and
then
just
give
you
the
quick
distribution
across
those,
possibly
a
thousand
rows.
So
you
can
do
a
quick
in
browser
analysis
alrighty.
So
let's
continue
with
this.
A
You
know
really
cost-effective
way
and
also
we
didn't
really
need
full-text
indexing,
because
we
thought
just
browser
like
in
browser
analysis
on
a
thousand
rows
should
be
enough
if
you
can
narrow
down
the
search
just
enough,
and
so,
let's
quickly
revisit
on
what
logging
and
we'll
of
kubernetes
looks
like
basically
we're
now
in
a
world
of
of
dynamically
shareable
workloads
that
have
that
have
job
labels
attached
and
but
sometimes
they
go
away.
The
the
pod
goes
away.
A
The
note
goes
away,
so
we
definitely
need
some
sort
of
log
irrigation,
because
when
the
note
goes
away,
you
probably
want
to
know
what
happened,
and
at
that
point
you
can't
anymore,
if
you
don't
have
a
prologue
irrigation
solution.
So
this
was
step
one
for
us
and
then
we
also
thought
there's
there
is
a
radio
system
out
there.
That
is
monitoring
really
well,
and
it's
dealing
with
this
dynamic
world
really
well
and
it's
prometheus
right
and
the
next
step
for
us
was
to
think
okay.
Well,
why
don't?
A
We
just
apply
those
same
principles
to
a
local
aggregates
solution
and
we're
the
final
logging
solution
that
that
that
I'm
scribing
here
it's
just
is
therefore
heavily
influenced
by
how
Prometheus
does
service
discovery,
how
to
find
all
the
targets
that
are
logging
and
also
it's
it's
taking
its
sins
from
on
how
labels
are
being
used
to
identify
log
streams
and
the
the
biggest
benefit
then
becomes.
If
you
are
consistent
with
your
Prometheus
logging
rules
and
you're
and
if
you're
applying
the
same
or
if
your
ever
feel
Prometheus
labeling
rules.
A
And
if
you
apply
the
same
laboring
labeling
rules
to
your
your
logs,
then
you
can
have
the
seamless,
switching
they're
showed
earlier
so
I
guess
I'm
introducing
Loki,
which
is
a
griffons
logging,
lock
regression
service.
It's
they
said,
there's
an
open-source
version
and
there's
a
hosted
version
and.
A
A
Well,
this
is
the
architecture.
So
central
is
the
logging
service,
the
Loki
service,
which
ingests
all
the
the
log
screams
and
on
your
node
you
would
need
to.
You
would
need
to
install
the
the
prom
tail
logging
agent.
That's
part
of
Loki
in
your
core
fauna
uses
at
the
locally
data
source,
and
when
you
have
all
these
three
components
in
place,
then
you
can
consume
your
logs
through
the
funnel
all
right.
So
the
open
source
version
is
available.
No
it's
not
available
there
because
they
need
to
release
it.
A
Cool
I'm
really
glad
that
worked.
So
sir
there's
also
a
hosted
version
because
we
thought
like
why
don't
we
just
run
this
stuff
ourselves?
Also,
so
you
don't
have
to
and
it's
an
alpha
but
we're
doing
an
all.
You
can
log
trial
whatever
that
means
and
which
is
going
to
be
free
until
February
or
something
or
no
q2
is
further
than
in
February
yeah.
So
for
a
long
time
or
until
it
blows
up
but
yeah,
we
courage
to
try
that
out
all
right.
So
one
caveat
there.
A
The
the
log
UI
that
you
saw
is
behind
a
feature
flag,
which
is
called
Explorer.
So
we
need
to
go
into
your
graph
on
our
config
uni
and
enable
it
there
and
but
in
February,
when
version
six
comes
out,
we're
gonna
release,
also
the
Explorer
UI,
so
that
you
don't
have
to
do
this
particular
step
anymore.
But
you
can
use
Loki
already
today
and
if
you
look
at
the
repo
there's
already
there's
also
a
command-line
client,
which
you
can
use
to
just
look
at
the
light
at
your
log
aggregation
yeah.
A
So
if
anything,
there
is
not.
You
know
if
you're
shaking
fists
again
just
write
to
me
first
yeah,
so
what
we,
what
we're,
still
working
on,
what
we
didn't
quite
get
to
was
to
integrate
tracing
with
this
whole
thing,
because
it
would
be
even
better
if
you
could
just
switch
not
just
from
metrics
to
logs
using
the
same
labels,
but
also
to
traces
so
and
we're
working
quite
hard
on
making
this
also
happen,
and
hopefully,
by
the
time
the
all
you
can
lock
trial
runs
out.
We
also
have
a
new
trial
for
tracing.
A
A
So
yeah,
so
the
question
was:
are
there
any
plans
to
integrate
with
other
logging
services
like
Splunk
or
yeah?
So
there
are
multiple
answers
to
this,
and
first
of
all,
graph
Anna
wants
to
be
the
the
the
or
wants
to
enable
the
existing
data
source
plugins.
Also
to
use
the
the
logging
user
interface
that
you
saw
so
we're
currently
building
an
app
and
like
a
logging
model
that
can
interact
with
these
right
so
that
you
can
get.
A
Yes,
oh,
no,
that
that's
actually
the
limitation
that
the
browser
or
or
that
we
set
in
the
browser.
Query
that
we
sent
to
the
back
end
that
we
that
we
think
is
a
nice
enough
data
set
to
do
in
browser
analytics
on
yeah.
But
you
can
set
this
to
whatever
you
want
right.
But
we
thought
it's
really
nice
because
you
usually
end
up
with
with
request
response
sizes
of
around
500
K
and
you
can
get
those
easily
over
the
wire
really
quickly.
A
Yeah,
so
we
don't
have
a
complete
answer
for
this,
but
for
example,
if
you
have
the
Alex
tech
already
and
you
were
we
will,
we
will
integrate
with
with
your
elasticsearch
back-end
on
the
user
interface
level
right
where,
where
we
just
send
an
an
elastic
query
that
comes
that
come
that
comes
back
with
logs
and
fields
and
it'll
be
transformed
into
our
internal
logs
model
in
the
browser.
And
then
you
can
consume
those
logs
there.
But
your
question
was
probably
more
around
fluency
and.
A
So
this
will
you
will
keep
writing
the
query
that
the
query
language
that
pertains
to
the
respective
data
source
so
yeah
there's
no
there's
no
abstraction
around
this
yeah
yeah.
So
we're
trying
to
keep
that
separate.
Like
the
the
interface
you
saw.
Has
those
query
rows
in
the
top
and
that's
going
to
be
determined
by
the
data
source
so
elastic,
for
example,
or
low
key
and
then
the
the
result
viewer
that's
going
to
be
the
shared
part,
yeah.
A
C
A
So
that's
a
question
you
have
to
answer
yourself.
Basically
Cabana
with
their
discovery,
for
example,
or
their
their
beats
system,
allows
you
also
to
to
frighten
Oscars
and
that
works
as
well
it,
and
it
has
a
lot
more
bells
and
whistles.
But
we
try
to
deliver
a
lot
like
this,
like
a
more
linear
system
with
I.
Guess,
that's
a
bit
more
straightforward,
at
least
that's
our
design
goal
yeah.
But
technically
you
can.
A
A
So
the
question
was:
is
there
going
to
be
a
nice
corner
operator
that
deploys
all
the
right
parts
right
to
kubernetes?
We
don't
really
have
one
right
now,
but
there
was
the
setup
slide.
That
gives
you
all
that
at
least
shows
you
what
we're
using
internally-
and
this
is
all
based
around
JSON.
It
are
using
JSON
it
okay,
yeah
yeah.
Well,
maybe
you
just
need
to
copy
and
paste
some
things
you
know,
but
I
can
yeah.
We
were
heavily
betting
on
JSON
it
in
case
on
it.
A
No,
no
yet
because
it
will
probably
be
something
that
is
similar
to
you
know
where
now,
when
I
clicked
on
no
no
hold
on
it's
it'll
be
it'll,
be
based
on
the
data
source
which
that
you
saw
at
the
top.
You
know
when
I
went
from
the
ops
tools
to
the
Loki
data
source,
and
the
idea
is
that
the
data
source
implements
their
own
importers
to
figure
out
which
labels
exist
on
the
on
the
data
source
that
I'm
about
to
choose
right.
This
is
how
the
currents,
which
is
implemented
yeah,.
B
A
A
A
No,
so
we
try
to
keep
it
really:
no,
no
frills.
We
we
have
spoken
to
customers
who
are
heavily
relying
on
this
because
they
are
a
bank
or
whatever,
and
but
for
us
this
introduces
a
lot
of
complexity,
that
we
don't
want
to
burden
ourselves
or
other
people
with
basically
yeah
at
this
point
yeah,
the
question
was:
will
there
be
per
lock
stream
permissions?
Well,
that's
a
question
right,
yeah
yeah
and
the
answer
is
not
full
of
the
foreseeable
future
yeah
or
maybe
I'm,
getting
it
I'm
gonna.
Maybe
here
yes,.
A
So
well
what
what
a
demo
mode
was
or
was
based
on
Prometheus
and
and
Loki
right,
and
you
can
use
similar
relabeling
rules
to
ensure
that
everything
is
consistent
there.
If
you're
our
story's
bit
less
clear
on
how
this
will
work
with
other
data
sources,
but
I'm
sure
it
will
be
documentation
about
this
yeah.