►
From YouTube: Traces from Events: A New Way to Visualise Kubernetes Activities - Bryan Boreham, Weaveworks
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon North America 2021 in Los Angeles, CA from October 12-15. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Traces from Events: A New Way to Visualise Kubernetes Activities - Bryan Boreham, Weaveworks
Many people know Distributed Tracing as a powerful way to visualise such complex operations, but as yet Kubernetes is not instrumented for tracing.
In this talk, Bryan will explore how Events, which are already emitted from all parts of Kubernetes, can be turned into OpenTelemetry Traces and visualised using Jaeger.
Outline of the talk:
- What are Kubernetes Events?
- What is Distributed Tracing?
- The Event data model
- Turning Events into Spans
- Reconstructing the parent-child relationships between Spans
- Demo
A
A
First
of
all,
let
me
introduce
myself.
I
work
at
weaveworks,
the
gitops
company.
I've
been
involved
in
open
source
software
for
decades
now
right
now,
I'm
a
maintainer
on
a
couple
of
cncf
projects.
Cortex
and
cni
worked
on
lots
of
other
ones
and
contributed
to
kubernetes,
but
what
I
really
passionate
about
is
is
understanding.
What's
going
on
inside
computer
systems,
I
a
lot
of
times.
I
I'm
I'm
saying
what
is
it
doing?
Why
is
it
so
slow
and
so
really
interested
in
tools
to
understand
those
things
and
get
a
better
idea
of?
A
What
we're
going
to
talk
about
what
are
events,
these
things
that
the
the
idea
is
based
on
what
are
traces
distributed
to
tracing?
What's
that
all
about,
and
I'm
going
to
show
you
a
program,
I
wrote
to
to
implement
this
idea
and
it's
called
k-span
if
you're
watching
this
talk
live
at
cubecon
any
questions
or
comments,
please
type
them
into
the
question
box.
I'm
there
watching,
along
with
you
and
I'll,
get
to
as
many
questions
as
I
can
at
the
end
of
the
talk.
A
So
I've
drawn
the
events
coming
from
different
kubernetes
components.
They
can
also
come.
These
are
supposed
to
be
your
your
pods
in
the
system.
Events
can
come
from
anywhere.
They
can
come
from
your
pods
from
from
operators
controllers
in
the
system
or
but
in
in
terms
of
what
I'm
demonstrating
right.
Now,
I'm
using
the
events
that
are
built
into
kubernetes
itself.
A
Each
event
has
a
type
which
is
which
is
either
normal
or
warning
has
a
kind
of
cryptic
reason
code.
It
is
about
some
object
within
the
system
I
mean
here.
The
first
few
are
about
a
pod.
Then
this
one's
about
a
replica
set
this
one's
about
a
deployment
and
then
there's
a
there's,
a
message
which
is
meant
to
be
more
human
readable.
A
If
we
drill
further
inside
I've,
I've
docked
out
the
contents
of
one
event
in
yaml
here
and
from
the
top
you
can
see
they
an
event.
Object
is
a
kubernetes
object,
so
it
has
an
api
version,
a
kind
it
has
a
name
and
namespace
and
the
metadata
these
are.
These
are
fields
you
find
in
pretty
much
any
kubernetes
object
and,
and
events
are
no
different
in
that
respect.
They
each
one
has
a
unique
name:
they
all
live
in
a
namespace.
A
Then
we
see
the
fields
I
was
talking
about.
The
the
type
the
same
event
might
happen
more
than
once,
and
kubernetes
will
count
them
up
for
you
and
and
tell
you
the
range
of
times
they
they
happened
over
this
this
one.
This
particular
one's
only
happened
once
so.
The
the
first
and
last
time
stamp
are
the
same.
A
The
message,
the
reason
we
talked
about
a
little
minute
ago,
and
each
one
also
has
a
a
component
that
it
comes
from
the
source
of
the
event
and
this
this
one
came
from
cubelet,
which
is
the
the
part
of
kubernetes
running
on
every
node
and
finally,
the
involved
object.
So
this
is
the
way
that
we
refer
to
another
object
in
kubernetes,
it's
quite
long-winded,
but
we
need
all
the
details.
We
need
the
name
and
namespace
of
the
object.
A
A
So
I'm
using
one
called
called
jager,
which
is
a
cncf
project.
There's
there's
a
few
other
ones
around.
I
think
zipkin
was
one
of
the
first.
There
are
commercial
products
like
honeycomb
and
lightstep,
so
there's
a
few
of
these
things
and
what
they
do.
They
collect
these
spans.
Think
of
think
of
a
time
span
with
a
beginning
and
an
end
and
something
that
happened
in
that
time
and
they
can
put
on
the
screen.
A
A
A
So
the
answer
is
yes,
I've
written
a
program
called
caseman
and
it
lives
at
the
weaveworks
experiments
repo,
because
it's
still
very
much
under
development
work
in
progress,
but
what
it
does
it
listens
to
events
and
it
emits
spans
using
the
open,
telemetry
protocol,
which
a
lot
of
distributed
tracing
systems
support.
Now
it's
takes
a
bit
of
work
to
do
match
up
all
the
events
inside,
so
we
can
figure
out
how
they
really
interrelate
to
each
other.
But
I'd
like
to
show
you
it
now.
Let's
take
a
look
at
the
demo.
A
I've
got
a
little
kubernetes
cluster
running
and
I'm
gonna
apply
a
manifest
in
so
brace
yourself.
Yaml
coming
here
we
go,
don't
be
scared.
All
it
is,
is
we're
gonna,
create
a
deployment
object
and
ask
it
to
create
two
pods
and
we're
just
using
a
demo
image
called
pod
info
from
from
my
colleague,
stefan
at
weaveworks,
but
it's
really
just
to
make
kubernetes.
Do
something
generate
some
events
and
see
what
we
can
do
with
the
visualization.
A
So
let
me
flip
over
to
my
terminal
window,
so
I'm
going
to
do
a
couple
of
things
here:
I'm
going
to
print
out
the
events
as
they
come
out
and
use
the
minus
w
for
for
watch
flag.
So
it's
going
to
sit
and
wait
for
any
events
to
come
along
and
on
the
other
window,
I'm
going
to
apply
that
manifest
that
we
just
looked
at
so
let's
see
what
it
does.
A
A
Now,
that's
the
the
text
coming
out
of
those
events.
I
was
running
k-span
in
the
background
this
whole
time.
So,
let's
flip
over
to
my
jaeger
window,
jager
has
this
cute
mascot
and,
let's
see
so,
let's
just
refresh
that
view.
Okay,
so
jager
has
now
heard
of
cubelet
and
I'm
going
to
ask
it
to
find
traces
relating
to
that
great
we've
got
one
trace.
A
traces
is
a
collection
of
spans
in
this
case,
got
14
spans
coming
from
what
is
that
five
different
components
in
the
system
just
a
minute
ago?
A
A
So,
let's
see,
let's
see
what
was
going
on
here.
The
whole
thing
took
nine
seconds
to
start
two
pods
we've
got
a
hierarchy
of
things
going
on
here
and
begins
with
an
update
to
the
deployment.
Well,
that
was
me
applying
that
manifest
and
sure
enough.
It
says
it
came
from
cube.
Cuddle
client
side
apply
next
thing.
That
happened,
something
in
the
deployment
controller.
A
We
can
click
in.
Take
a
look
at
that
and
we've
got
some
more
detail
here.
So
k-span
has
pulled
out
these
details
on
the
event,
particularly
the
message.
So
first
thing
was
it:
it
scaled
up,
the
replica
set,
so
that's
great
replica
set.
This
is
just
the
way
that
kubernetes
manages
pods
in
a
deployment,
and
you
know
you.
A
So
it
took
a
few
seconds
to
pull
the
image
down
over
the
internet
and
that's
that's
a
very
typical
use
of
distributed.
Tracing
ui
is
to
show
the
bit
that
took
the
longest
time
was
pulling
the
image,
also
that
it
pulled
it
twice
for
two
pods,
so
it
pulled
the
image.
That's
when
it
finished
pulling
it
then
it
created
the
container.
Then
it
started
the
container
and
those
things
were
very
fast
and
it
did
two
pods,
which
is
what
we
wanted.
A
A
A
A
A
A
A
A
A
Let's
go
back
to
the
architecture
diagram,
so
events
come
in,
but
they
don't
come
in
in
the
perfect
order.
It's
a
it's
a
distributed
system.
These
events
are
being
generated
from
all
over
it
and
it's
very
much
in
the
nature
of
distributed
systems
that
things
will
apply
will
arrive
in
an
order
that
may
not
be
convenient
to
you.
A
So
first
thing
we
do
is
we
stack
up
events
that
have
come
in
on
the
basis
that
we
might
not
be
able
to
make
sense
of
them
yet
each
one
as
it
comes
in?
We
do
that
process
of
fetching
the
ownership
chain
that
I
just
showed
you
and
we
end
up
with
a
collection
of
all
these
things.
Now,
let
me
stress,
k-span
is
not
coded
well.
98
of
it
is
not
coded
to
understand
things
like
deployments
and
replica
sets.
A
So
so
we
end
up
with
this
kind
of
collection
of
objects
and
things
which
happened
recently,
which
are
sometimes
pending
events,
and
then
at
some
point
say
when
we
walk
to
the
top
of
the
chain,
we
figure
out
that
we've
got
a
whole
new
operation,
so
we
start
a
new
trace,
and
now
we
can
start
assigning
spans
to
these
events
and
generating
it
spans
to
emit
from
the
program
now.
The
last
thing
that
happens,
we're
not
quite
done
yet.
A
A
So
in
order
to
generate
the
the
spans
that
you
saw
on
screen
with
a
longer
length
of
time,
what
we
do
basically
is
is
watch
first,
every
time
something
new
happens
on
an
object.
We
call
that
the
end
of
the
previous
one
and
we
mark
the
beginning
and
end
times
that
way,
the
the
it's
kind
of
fudging
it,
but
it
makes
for
a
much
better,
more
understandable
display,
and
you
saw
things
like
the
the
pulling
of
the
image.
There
is,
in
fact,
an
event
at
the
start
and
end,
so
we
can.
A
So
we
we,
we
kind
of
hold
them
at
the
beginning
to
match
up
to
find
out
how
they
relate
to
other
objects,
and
then
we
hold
them
just
after
we,
we
generated
the
spans
because
we
want
to
put
the
end
time
on.
We
don't
hold
them
indefinitely.
If,
if
nothing
happens
for
a
few
seconds,
we
we
send
it
off.
It's
we're
trying
to
do
a
reasonable
job,
trying
to
balance
having
a
a
gui,
that's
reasonably
up
to
date,
with
trying
to
put
the
accurate
information
on
on
the
object.
A
A
A
A
A
more
tailored
gui,
by
which
I
mean
we
could
draw
this
in
a
way
that
that
makes
more
sense
for
kubernetes
events,
the
generic
display
of
jaeger
and
other
distributed
tracing
systems
is
oriented
in
a
particular
way
towards
analyzing
transactions
in
a
system.
So
probably
there
are
ways
to
put
this.
This
kind
of
kubernetes
information
about
deployments
and
things
like
that
on
screen
in
in
a
more
focused
and
relevant
way.
If
we
get
some
talented
designers
on
the
job.
A
And
lastly,
what
is
upstream
of
the
point
where,
where
something
happens
inside
kubernetes,
so
if
you
think
about
rolling
out
a
new
version
of
of
the
software,
maybe
this
maybe
something
happens
in
your
ci
cd
system.
You
could
get
spans
from
there
and
link
them
all
together.
I
work
for
waveworks
the
gitops
company,
so,
of
course
we
would
expect
the
moment
at
which
a
change
is
applied
to
kubernetes
to
come
from
a
git
commit.
So
that
would
be
this
operation
and,
and
how
did
we
get
here?
Did
we
go
through
a
workflow?
A
A
Well,
thank
you
for
watching
we'll
flip
to
question.
Answering
live
in
just
a
minute.
So
if
you
do
have
any
questions,
you
still
have
time
to
type
them
in
and
I'll
just
leave
you
with
a
few
links
there.
The
code
for
the
case
ban
is
there
at
github
weave
works,
experiments
open
telemetry,
which
is
the
protocol
linking
everything
together
and
jaeger
the
distributed
tracing
system
I
used
in
the
demo.