►
From YouTube: Keynote: ...What Does the Future Hold for Observability? - Tom Wilkie & Frederic Branczyk
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Keynote: Metrics, Logs & Traces; What Does the Future Hold for Observability? - Tom Wilkie, VP Product, Grafana Labs & Frederic Branczyk, Software Engineer, Red Hat
The “the three pillars” of observability - monitoring, logging and tracing - were so 2018. A growing consensus is questioning the efficacy of this approach. So what’s the next step?
Without dwelling on past, Frederic and Tom will present three exciting trends in observability for 2019 and beyond. Future developments need to tame complexity, enhance understanding and accelerate incident response.
Expect substantiated opinions, contentious observations and surprising predictions. Are you ready to take this next step?
https://sched.co/MRzA
A
So
today
we're
going
to
talk
about
observability
and
what
Fredrik
and
I
see
as
the
future
or
potentially
the
future.
Hopefully,
everyone
here
is
familiar
with
the
term
observability
yes,
No,
okay,
so
observability
can
normally
do
be
defined
as
these
kind
of
three
pillars:
metrics
logs
and
traces.
This
is
a
mental
model
for
people
who
are
maybe
new
to
the
space
and
and
helping
to
get
a
handle
on
what
they
need
to
do
and
what
they
need
to
implement.
A
I'm
just
going
to
give
you
a
quick
overview
of
what
they
are
and
the
projects
in
this
space
metrics
normally
time
series
data-
this
is
kind
of
if
you're
doing,
trends
of
memory,
usage
or
latency
and
the
CNCs
has
some
great
projects
in
this
space.
Open
metrics
is
a
exposition
format,
so
a
format
for
exporting
metrics
from
your
application
and
Prometheus
is
probably
now
the
de-facto
monitoring
system
for
for
kubernetes
and
apps
on
kubernetes,
so
logs.
Sometimes
events
in
kubernetes.
This
would
be
what
comes
out
stood
out
on
your
containers.
A
In
this
space,
the
CN
CF
has
fluently
log
shipping
agent
and
then
traces.
This
is
potentially
the
hardest
one.
To
sum
up
in
a
single
sentence,
so
we're
trying
to
find
something
short
and
catchy
so
I
think
of
distributed
traces
as
a
way
of
recording
and
visualizing
a
request
as
it
traverses
through
the
many
kind
of
services
in
your
application.
B
A
So
in
this
talk,
Frederic
and
I
are
going
to
present
three
predictions.
It's
always
a
bit
of
a
risky
business.
Giving
predictions
will
probably
be
wrong.
We're
gonna
give
it
a
go
anyway.
So
first
prediction
there
will
be
more
correlation
between
the
different
pillars.
We
think
this
is
the
year
when
we're
going
to
start
breaking
down
the
walls
and
we're
going
to
start
seeing
joined
up,
workflows.
A
I
want
to
talk
about
three
examples
of
what
you
can
do
right
now,
and
hopefully,
after
the
talk
you
can
come
to
me
with
more
examples
and
in
the
next
few
years
we
can.
We
can
make
this
really
happen,
so
metrics,
lobs
and
traces.
The
three
examples
I've
got
are
being
able
to.
You
know,
look
at
a
graph
of
let's
say
your
server
latency
and
instantly
being
able
to
switch
to
the
logs
relevant
for
that
service.
A
The
second
example
would
be
when
looking
at
those
logs
finding-
let's
say
the
slowest
query-
that's
causing
that
peak
in
latency
that
you're
diagnosing
being
able
to
instantly
jump
to
the
trace
for
that
request
and
see
where
it
spent
time
throughout
your
architecture,
and
the
final
example
is
similarly
looking
at
that
latency
graph
wanting
to
jump
straight
to
the
trace
so
the
first
system.
This
is
actually
a
project
that
I
work
on
myself
is
called
Loki.
This
is
an
open
source
log
aggregation
system
by
graphing.
A
We
launched
it
at
coop
con
in
Seattle
six
months
ago,
had
an
absolutely
great
response,
like
loads
of
people
have
given
us
really
good
feedback.
Loki
uses
Prometheus's
service
discovery
to
automatically
find
the
jobs
within
your
cluster.
It
then
takes
that
service
discovery
and
the
labels
that
that
service
discovery
gives
you
and
associates
them
with
the
log
stream,
and
it's
this
kind
of
systematic,
consistent
metadata.
That's
the
same
between
your
logs
annual
metrics
that
enables
that
switching
between
the
two
seamlessly.
A
So
that's
the
first
example.
The
second
exam
Paul
is
elastic,
search
and
Zipkin,
so
elastic
search
is
probably
the
most
popular
log
aggregation
system.
Even
I
can
admit
that
and
Zipkin
is
probably
the
open.
The
original
open
source
distributed
tracing
system
there's
in
Cabana,
which
is
the
elastic
search
UI.
There
is
a
thing
called
filled
for
matters,
and
some
chap
on
Twitter
set
up
his
Cabana
to
insert
a
link
using
a
field
formatter
so
that
he
could
instantly
linked
to
his
Zipkin
traces.
I
think
this
is
really
cool.
A
I'm,
really
looking
forward
to
adding
this
kind
of
feature
to
grow
fauna.
So
then
the
third
example
would
be
open
census
or
now
open
telemetry.
Of
course
they
have
a
thing
called
exemplars.
This
is
where
a
few
example
trace.
Ids
are
associated
with
every
single
bucket
in
a
histogram,
and
then,
when
you
go
to
that
bucket-
and
you
can
see
you
know
what's
causing
this
high-latency,
you
can
link
straight
to
the
the
trace.
A
I
think
you
know,
I,
like
that
open
census
and
open
telemetry
is
open
source
I'm,
not
actually
yet
aware
of
an
open
source
system
on
the
server
side.
That's
actually
implemented
this
workflow.
If
I'm
wrong
come
and
find
me
afterwards-
and
let
me
know
so-
that
was
three
examples
of
what
you
can
do
right
now,
how
you
can
do
this
correlation
that
I'm
talking
about
I'm,
really
looking
for
more
examples
and
looking
at
ways
in
which
we
can
make
this
more
automatic.
What
do
you
think
I
think.
B
What
you
said
about
metadata
is
exactly
the
way
that
this
is
going
to
work.
A
I
think
metadata
is
what's
going
to
allow
us
to
do
all
of
this
correlation,
but
still
be
able
to
build
systems
that
are
useful
individually,
but
also
as
a
whole,
and
do
do
all
this
great
correlation
across
the
signals.
So
that's
prediction
number
one
and
prediction
number
two:
is
we
believe
that
we
are
going
to
be
seeing
a
lot
more
signals
and
new
analysis
now
as
humans?
B
We
like
the
number
three,
that's
why
maybe
we've
settled
on
the
three
pillars
of
observer
ability,
but
we
believe
we
believe
there
is
so
much
more
data
that
can
help
us
have
insight
into
our
and
for
our
running
systems,
and
but
it
doesn't
stop
with
data.
We.
When
we
have
this
data,
we
actually
need
to
do
something
useful
with
it
right.
So
it's
signals
and
analysis
that
are
going
to
bring
us
forward
and
I
want
to
show
you
a
concrete
example
of
what
maybe,
in
the
future,
we'll
be
calling
the
fourth
pillar
of
observability.
B
So
what
I've
got
over
here
is
a
graph
of
memory
that
we're
that
I've
created
using
Prometheus
and
what
we're
seeing
is
memory
usage
over
time
and
we
see
a
sudden
drop
of
memory
and
then
we're
seeing
a
new
line
that
has
a
different
color
and
in
prometheus.
This
means
is
a
distinct
time
series,
but
we're
actually
looking
at
the
same
workload
here.
B
So
what
we're
seeing
in
this
graph
is
actually
what
we
call
an
in
kill
so
when
our
our
application
has
allocated
so
much
memory
that
the
current
has
said,
stop
here,
I'm
going
to
kill
this
process
and
go
on.
So
that's
that's.
What's
called
an
M
and
our
existing
systems
can
show
us
all
of
this.
All
this
data,
all
this
usage
over
time
and
our
logs
can
tell
us
that
the
M
has
happened.
But
when
we're
the
developer
of
an
application
like
this,
where
this
happens,
we
actually
want
to
know.
How
can
we
fix
this?
B
So
it
turns
out
this
is
not
an
entirely
new
space
and
Google's
has
published
a
number
of
papers
white
papers
on
this,
and
there
are
some
proprietary
software
that
do
this,
but
it
hasn't
really
been
solved
in
the
open
source
space.
So,
let's
imagine
for
a
second.
If
we
had
a
Prometheus
like
system
that
periodically
took
memory
profiles
of
our
application,
then
we
would
kind
of
be
creating
memory
profiles
over
time.
B
We
could
maybe
even
call
it
a
time
series
of
profiles
and
then,
if
we
had
taken
every
10
seconds
or
every
15
seconds
a
memory
profile
of
our
looming
application,
maybe
we
would
actually
be
able
to
figure
out
what
has
caused
this.
This
particular
incident,
so
it
turns
out
I've,
actually
looked
into
this
space
and
I
would
I've
started
a
new
project
called
comm
prof,
continuous
profiling,
because
I'm,
not
a
very
imaginative
person
and
I'd
like
to
invite
everybody
to
help
me
and
explore
this
space
together
with
me.
B
Now
I've
only
mentioned
how
we
can
collect
this
data,
but
how
can
we
make
even
more
use
out
of
this
data
than
just
looking
at
the
normal
memory
profile?
So
I've
got
a
P
prof
profile
over
here.
That's
basically,
what
the
go
runtime
gives
us
to
analyze
our
our
running
systems
and
as
it
turns
out
as
I,
was
putting
together.
These
slides
I
found
a
memory
leak
in
con
prof,
so,
but
how
about
if
Khan
prov
could
have
told
me
which
part
of
contra
is
X
actually
has
a
memory
leak
right.
A
The
well-known
tradition
of
reimplemented
Google
ideas
and
open
source
excellent,
so
prediction
number
three:
the
rise
of
index
free
log
aggregation
we've
been
seeing
this
to
develop
over
the
last
well
six
months
to
a
year
or
so
a
lot
of
people
have
been
saying
things
like
just
give
me
log.
Just
give
me
grep.
Sorry,
just
give
me
log
files
and
grep
I,
think
you
know
the
systems
like
spunk
and
elasticsearch.
A
You
know
they
give
us
a
tremendous
amount
of
power
to
search
and
analyze
our
logs,
but
all
of
this
power
comes
with
a
lot
of
I'm,
not
going
to
say
responsibility.
It
comes
with
a
lot
of
complexity
and
expense,
and
before
we
had
Splunk
and
elasticsearch
logs
were
just
stored
as
files
on
disk,
maybe
in
some
centralized
server
somewhere
and
you
would
just
go
on
and
grep
it
and
I
think
we're
starting
to
see
the
desire
for
simpler
index
free
log
aggregation
systems.
A
You
know
effectively,
everything
old
is
new
again
so
I'm
going
to
give
you
three
examples
again.
Just
three
is
a
very
aesthetically
pleasing
number.
The
first
example
is
a
project
called
ok
log
by
someone
called
Peter
Balkan
started
this
project
I
think
just
over
a
year
ago.
Unfortunately,
it's
been
discontinued,
but
it
had
some
really
great
ideas
about
distributing
grep
over
a
cluster
of
machines
and
being
able
to
basically
just
brute
force
your
way
through
your
logs
made
it
a
lot
easier
to
offer
a
lot
cheaper
to
run.
A
Everyone
knows
cube
catalogues,
I,
think
if
we
squint,
we
can
think
of
this.
As
a
log
aggregation
system,
you've
got
a
central
place,
you
can
go
and
query
your
logs.
It
goes
and
stores
them
in
a
distributed
way.
I
think
you
know
for
me
the
thing
that
was
really
missing
from
coop
catalogs
is
being
able
to
get
logs
for
pods
that
are
missing,
pods
that
disappeared
or
pods
that
earned
or
failed,
especially
during
rolling
upgrades,
and
that's
really
why
I
started
locally
so
again.
Low
key
second
mentioned
sorry
about
that.
A
Loki
is
a
index
free
log
aggregation
system
designed
to
be
easy
to
run
an
easy
to
scale.
It
doesn't
come
with
the
power
of
something
like
elasticsearch.
You
wouldn't
use
loki
for
business
analytics,
but
loki
is
really
there
for
your
developer,
troubleshooting
use
case
so
I'm
really
hoping
that
2019
and
2020.
We
see
the
rise
of
these
index.
Free
developer,
focused
log,
app
log,
aggregation
systems
and
I'm
hoping
this
means
as
a
developer.
I'll
never
be
told
to
stop
logging
so
much
data
again.
A
So
that's
our
three.
That's
our
three
predictions
more
correlations
between
pillars,
new
signals
and
new
analysis
and
the
rise
of
index.
Three,
the
log
aggregation.
These
are
just
three
ideas.
You
know
if
I
think,
if
we're
lucky,
we
can
watch
this
talk
in
a
year
or
two
and
maybe
get
one
out
of
three
but
yeah.
What
do
you
think
I.
B
B
Observability
was
not
met
created
by
a
few
people.
This
it
was
created
by
it,
was
created
by
people
who
had
lack
of
tooling
in
their
troubleshooting.
So
I
want
everybody
in
this
room
to
go
and
think
next
time
as
you're
troubleshooting,
something
and
you're
leaving
your
existing
tooling.
What
data
are
you
looking
at
and
what
are
you
doing
to
troubleshoot
your
problem,
and
can
we
do
that
in
a
systematic
way?