►
From YouTube: SIG Instrumentation Introduction And Deep Dive - Damien Grisonnet, Red Hat; Han Kang & David Ashpole
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe in Amsterdam, The Netherlands from April 17-21, 2023. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
SIG Instrumentation Introduction And Deep Dive - Damien Grisonnet, Red Hat; Han Kang & David Ashpole, Google
Speakers: Han Kang, David Ashpole, Damien Grisonnet
Kubernetes SIG Instrumentation is responsible for ensuring high quality and consistent instrumentation across the Kubernetes project. We will begin with an introductory overview of the efforts the SIG Instrumentation has worked on in the past and is currently working on. This deep dive session will go into detail about currently ongoing efforts happening within SIG Instrumentation to share with the audience concrete pieces of work to encourage future collaboration. Software engineering and operations are both disciplines practiced in SIG Instrumentation, and any experience will help the special interest group's mission. Join this session to learn how to get involved in SIG Instrumentation to make instrumentation even better!
A
C
C
B
B
First,
let's
go
over
the
charter
yeah.
The
charter
is
basically
to
cover
the
best
practices
for
observability
across
kubernetes
components.
So
basically
this
means,
while
we
don't
own
individual
metrics,
we
own
the
practices
and
sort
of
the
format
and
the
tools
that
people
use
to
instrument
kubernetes
components
like
the
cube,
API
server,
the
scheduler
controller
manager.
All
of
these
things
that
you
are
familiar
with,
we
have
a
large
number
of
sub-projects.
B
B
We
also
work
a
lot
on
metrics,
so
Prometheus
metrics,
which
are
instrumented
across
all
of
our
kubernetes
components
and
logs,
which
are
also,
in
all
kubernetes
components,
we're
working
on
structured
logging.
That
way,
you
can
get
you
know
structured
format
instead
of
dealing
with
freeform
text
and
events,
as
well
as
traces.
B
C
So
crypto
metric
is
a
simple
Prometheus
exporter,
which
is
like
a
piece
of
software
that
is
able
to
convert
any
third-party
data
into
primitive
into
metrics
in
the
promises
format
and
in
the
case
of
keep
State
metric.
We
are
getting
metrics
from
all
the
kubernetes
objects
in
general,
since
those
can
be
quite
insightful
in
the
cluster.
But
the
reason
why
this
is
not
part
of
kubernetes
directly
is
because
the
metrics
can
be
quite
variable,
and
maybe
out
of
scope
for
like
the
normal
kubernetes
setup.
C
When
I
don't
know
a
deployment
is
failing
or
whatever,
and
some
of
the
example
that
we
have
is
that
the
cube
deployment
spec
replica,
which
allows
you
to
know
at
one
point
in
time
how
much
replica
your
deployment
had,
as
well
as
the
studies
of
the
replica
dated,
which
means
that
if
there
was
some
failures
during
a
running
update
or
whatever
like,
you
would
be
able
to
know
it
via
this
metric.
So
it's
really
insightful
to
investigate
some
values
and
how
this
keep
State
metric
does.
C
That
is
so
cubesat
magic
is
watching
for
events
on
GPI
server
on
a
regular
basis
and
whenever
there
is
a
new
object
that
is
created,
updated
or
that
there
is
any
like
mutation
towards
an
object
on
kubernetes
cubesat
Metric
will
get
that
information
and
convert
it
into
a
metric.
That
would
then
be
able
to
be
collected
by
Primitives,
and
you
would
be
then
able
to
create
others
on
it.
C
Another
project
that
is
that
you
may
not
have
heard
of
but
like
might
be
running
a
new
cluster
because,
like
it's
essential
nowadays
in
a
kubernetes
cluster
which
is
metric
server,
it
implements
one
of
the
API
that
we
support
as
a
group,
which
is
the
resource
metrics
API,
which
is
like
the
metric
that
is
under
command,
such
as
cube
cattle
top.
That,
essentially
like
allows
you
to
know
the
resource,
your
usage
of
your
pods
and
nodes
as
well,
and
some
other
and
other
use.
Cases
of
this
API
is
for
auto
scaling.
C
What
it
does
is
that,
well,
there
is
some
limitation
when
you
are
doing
Auto
scaling
with
metric
servers,
since
you
only
have
like
usage
metrics
and
you
might
want
I,
don't
know
like
Toyota
scale.
Your
application,
based
on
the
number
of
requests
that
are
currently
being
processed
by
your
application,
and
that
is
done
via
primitive
software,
for
example.
But
there
are
many
other
adapters
as
long
as
they
support
the
custom
or
external
Matrix
API.
They
should
be
able
to
Auto
Scale
based
on
any
kind
of
signal
and
in
the
case
of
primitive
setup.
C
What
it
does
is
that
when
it
receives
a
call
from
the
API
server
to
get
like
I,
don't
know
metrics
about
like
the
number
of
item
in
your
rabbit
mqq,
and
then
it
will
convert
that
request
to
a
prompt
code.
Query
and
query
Primitives
that
and
create
the
Primitive
server
directly
and
return.
You
the
results
so
that
your
auto
scaling
pipeline
will
be
able
to
directly
like
contact
from
this
adapter
to
get
any
kind
of
information
that
would
be
related
to
your
application.
C
So
now
that
we
gone
through
like
a
couple
of
the
project
that
we
own
I
will
go
through
the
like
we,
as
I
mentioned
in
the
beginning,
we
own
the
three
major
obserability
signal
as
part
of
our
Sig
and
I,
will
now
go
through
the
logs.
So
with
the
logs,
like
we
own
the
infrastructures,
around
logging
kubernetes,
as
well
as
some
parts
of
the
logging
output
that
is
produced
by
all
the
kubernetes
components
like
we
don't
hold,
what
the
other
group
are
doing
with
the
logs
and
how
they
produce
it.
C
But
we
are
responsible
for
helping
them
writing
logging
into
their
code,
as
well
as,
like
writing,
a
good
on
how
to
write
a
good
log
line
in
general,
and
we've
invested
quite
a
lot
the
past
couple
of
years
in
logging,
because
well
before
we
had
no
real
structure
in
our
login.
The
format
was
essentially
iterated
from
the
Kellog,
which
is
like
the
logging
tool
that
we
are
using
to
produce
log
all
of
the
kubernetes,
and
it
was
just
a
basic
string.
C
C
The
first
one
is
text
based
which
is
mostly
used
by
developers,
because
it's
quite
easy
to
go
through
and
being
able
to
investigate.
It
is
way
simpler
than
Json,
and
then
we
also
have
Json,
which
is
what
most
of
the
logging
platform
today
ingest
and
then
it's
easier
for
this
platform
to
build
querying
on
top
of
these
log
lines
since,
like
Json,
is
more
formatted
in
general.
So
that's
what
we
have
worked
on
for
the
past
couple
of
releases.
C
C
C
If
you
try
it,
if
you
want
to
have
very
specific
behavior
depending
on
the
scenario
you
are
in,
so
what
we
did
is
that
now
we
will
pass
the
logo
via
the
the
context,
which
means
that
along
your
call
chain,
you
will
be
able
to
modify
the
the
logger
and
add
some
new
information
to
it.
The
most
common
use
cases
that
we
would
have
is
to
make
sure
that
once
you
reach
a
certain
point
in
the
cold
chain,
you
had
a
key
value
pairs
to
all
the
logs
that
will
be
produced
via
this
context.
C
C
Just
by
enforcing
this
particular
level
and
value
to
all
the
logs
and
yeah,
this
is
implemented
by
a
new
API
in
Kellogg
that
was
introduced
by
the
group,
and
there
is
like
many
card
changes
that
will
need
to
be
made
in
order
to
spread
that
over
all
documentaries
component,
and
we
really
need
like
some
additional
contribution
in
that
for
that
part
of
the
code.
Yes,.
C
Okay
and
and
we
will
need
additional
contributors-
the
working
group
structure,
logging,
so
I'm
talked
about
the
groups
instrumentation,
but
in
kubernetes
there
is
also
the
working
group
which
has
like
kind
of
smaller
groups
that
are
like
related
to
a
Sig
but
are
very
dedicated
to
one
particular
area
of
the
code
like
in
the
case
of
structure
logging.
It
was
first
meant
to
introduced
a
way
to
build
structure,
logging
kubernetes,
but
now
it
has
a
bit
bigger
purpose,
since
it's
also
had
to
cover
like
contextual
logging,
there
are
two
organizers,
Eric
and
Patrick.
C
You
can
reach
out
to
them
via
this,
their
stock
Channel
and
you
can
go
through
also
the
bi-weekly
meetings.
If
you
want
to
contribute-
and
this
is
one
of
the
best
area-
to
contribute
I
guess
as
a
new
kubernetes
contributors,
because
you
will
be
able
to
discover
very
various
part
of
the
code
base
since,
like
everything
needs
to
be
changed
in
terms
of
logging,
you
will
be
able
to
have
your
hand
into
many
aspects
of
the
code
base,
and
it's
also
like
well
documented
how
to
do
the
change.
D
D
So
there's
a
thing
called
a
trace
context:
that's
propagated
between
components
in
your
microservice
architecture,
and
that
has
a
trace,
ID
and
span
ID,
and
with
that
each
component
can
produce
Telemetry
that
we
call
spans
that
have
a
single
unifying
attribute
across
all
of
them,
and
that
way
some
Trace
backend
can
stitch
those
all
together
into
a
graph.
Like
you
see
there
below.
D
D
D
D
Okay,
so
let's
dive
a
little
bit
deeper
into
API
server
tracing
because
that's
been
a
fun
project.
I've
been
working
on.
So
this
is
what
it
initially
looked
like
in
Jaeger,
which
is
another
cncf
project.
As
of
kubernetes
122.,
it's
pretty
Bare
Bones,
like
you
can
see.
The
top
span
is
for
the
user's
call
to
the
API
server,
an
API
server
serving
that
and
then
the
middle
one
is
the
API
servers
at
CD
client,
making
a
request
at
CD
and
the
bottom
one
is
actually
coming
from
FCD
and
is
responding
to
the
API
server.
D
That
sort
of
looks
kind
of
like
a
trace
and
tells
you
you
know,
sort
of
what
happened,
and
so
one
of
the
improvements
that
landed
I
think
yesterday,
is
that
we
were
actually
able
to
sort
of
sneak
in
and
change
those
calls
to
also
integrate
with
open
Telemetry
based
tracing.
So
now
you
can
see
the
red
boxes
there,
highlighting
some
of
the
new
things
that
have
come
from
using
existing
text-based,
instrumentation
and
migrating
that
to
also
use
open,
Telemetry
and
actually
I
have
a
really
quick
demo.
D
So
first
I'm
going
to
start
up
start
up
a
Jaeger
pod
locally
and
second
I've
written
this.
It's
actually
just
a
modified
version
of
the
API
server
integration
test,
but
basically
I'm
going
to
configure
API
server
tracing
like
so
and
set
it
to
100
sampling
rate
I'm,
going
to
pass
that
to
it
and
then
I've
also
turned
on
at
CDs
tracing
here.
Just
for
the
integration
test
and
I've
got
it
going
to
a
local
otlp
endpoint,
which
is
supported
by
Jaeger.
D
C
D
D
D
And
then
I'll
click
on
the
first
result
and
there
we
go
so
this
is
actually
the
same
as
that
screenshot,
but
it's
much
more
fun,
because
I
can
actually
go
in
and
look
at
all
the
interesting
attributes
and
stuff
that
have
been
added
here.
All
that
actually
existed
in
API
server
code
probably
has
existed
for
years,
but
hopefully
is
now
much
much
more
usable.
D
Okay,
cool:
let's:
let's
talk
briefly
about
the
cubelet
tracing,
so
this
is
a
cubelet
trace
from
the
trace
feature
that
landed
in
125.
and
it's
pretty
basic.
To
be
honest,
so
what
you
can
see
is
the
top
line
is
from
the
cubelet,
and
that
is
when
the
cubelet
started
the
request
to
create
a
container,
and
then
the
bottom
line
is
from
the
container
runtime,
which
is
where
it's
serving
the
request
to
create
a
container.
D
D
One
of
the
things
that's
missing
from
the
cubelet
is
that
we'd
really
love
to
have
some
way
to
tie
together
a
whole
bunch
of
container
runtime
operations
like
in
order
to
create
a
pod.
I
need
to
make
a
Sandbox
I
need
to
pull
images.
I
need
to
create
containers
right
and
I'd
like
to
see
how
all
those
are
related.
B
B
B
So
there
was
this
one
time.
I
was
on
call
and
basically
I
was
debugging.
Some
issue
and
I
had
to
look
at
charts
except
one
of
the
charts
that
I
was
wanting
to
look
at,
had
no
data
or
it
was
blank
and
I
looked
at
the
underlying
data
and
there
was
no
data
and
after
doing
a
bit
of
digging,
it
turned
out
that
open
source
or
you
know,
Upstream
kubernetes-
had
basically
renamed
a
metric.
B
So
I
don't
know
how
much
you
guys
have
thought
about
this,
but
one
cannot
simply
rename
metric
because
when
you
rename
a
metric,
the
metric
actually
just
ceases
to
exist
right.
You
end
up
with
a
new
metric,
so
all
of
all
of
your
charts,
which
reference
a
metric
by
name.
No
just
don't
do
anything
right,
and
so
there
was
this
other
effort
going
on.
At
the
same
time
called
the
metrics
overhaul
and
what
metrics
overhaul
was
doing.
B
B
Basically,
something
called
the
kubernetes
metrics
framework,
and
what
this
does
is
this
introduces
stability
classes
to
metrics
and
allows
us
to
guarantee
a
certain
API
around
metrics
right.
So
how
did
we
do
this
well,
like
Prometheus?
Doesn't
natively,
allow
you
to
add
stability
levels
to
metrics
or
metric
definitions?
So
we
wrapped
all
of
the
Prometheus
descriptors
and
we
added
a
metadata
field
called
stability
level,
and
so
you
can
annotate
metrics
with
a
stability
level,
and
then
we
created
a
static
analysis
pipeline.
B
So
I,
don't
know
how
familiar
you
guys
are
with
static
analysis.
It's
basically
like
compiler
level
stuff
where
we
go
through
the
entire
kubernetes
code
base.
We
analyze
it.
We
look
for
all
metric
definitions
and
we
look
for
stability
classes.
Then
we
parse
that
information
and
we
guarantee
that
the
structure
for
a
stable
metric
does
not
violate
the
guarantees.
B
We
provide
the
community,
and
so
we
have
built
that
we
have
been
improving
on
that
over
time
and
not
only
that,
but
this
requires
a
lot
of
code
right
like
we
don't
want
to
break
the
entire
world's
ability
to
operate
kubernetes.
That
would
be
terrible
and
we
basically
nearly
did
that
with
the
metrics
overhaul.
B
B
We
are
extending
this
framework,
so
when
we
started
this
framework,
we
actually
only
had
two
stability
levels
we
had
stable
and
we
had
Alpha
and
stable
metrics
had
guarantees
to
not
change
for
at
least
one
year
right,
so
you
can
guarantee
that
your
charts
alerts
will
not
break
for
a
year.
Alpha
metrics
don't
have
any
such
guarantees.
We
are
now
introducing
beta,
which
is
experimental.
We
have
not
yet
defined
what
guarantees
these
have
and
we
are
working
on
this
and
we
would
love
Community
input
on
what
would
be
reasonable
for
a
beta
class
metric
right.
B
We
also
have
internal
metrics.
These
are
likely
to
not
have
stability
guarantees,
some
metrics
basically
describe
code
or
in
implementation,
detail
of
kubernetes
and
therefore
changes
when
the
code
changes
right.
So
these
are
like
highly
variable
mutating
type
metrics.
Those
would
be
classified
as
internal
would
probably
not
have
stability
guarantees
and
you
probably
don't
even
want
to
chart
them
or
Define
slos
for
them.
B
The
static
analysis
pipeline
that
we
built
basically
parses
the
entire
kubernetes
code
base
all
over
the
metrics
and
it
basically
allows
us
to
Auto
generate
documentation
for
every
single
kubernetes
metric
defined
in
the
code
base,
and
this
basically
recently
landed
and
in
126
we
will
have
documentation.
You
will
be
able
to
see
every
single
metric
that
the
kubernetes,
the
core
kubernetes
control,
plane,
outputs,
the
scheduler,
the
cubelet
API
server
controller
manager.
B
B
So
if
you
want
to
contribute
come
to
our
meetings,
who
we
are,
we
are
like
you
know,
pretty
friendly,
open,
open,
Sig
and
I
think
one
of
the
easier
things
to
get
involved
with
yeah.
It's
a
little
bit
easier,
I
think
for
newcomers,
so
we
would
love
for
you
guys
to
get
involved
yeah.
You
should
come
to
our
Sig
meetings.
We
have
a
Sig
meeting
every
week.
B
We
welcome
you
to
participate,
we're
also
looking
for
contributors
for
our
sub-projects,
like
Cube,
State,
metrics
and
Prometheus
adapter,
and
in
fact
we
have
been
discussing
deprecating
some
of
our
sub-projects
because
of
a
lack
of
contributors
like
basically,
we
can't
support
the
large
number
of
sub-projects
that
we
actually
own
and
we
are
in
dire
need
of
people
to
come
and
help
us
with
it.
So
please
please,
come
metrics
server
and
structured
logging
are
also
in
need
of
contributors
and
you
can
contact
Merrick.
B
B
We
also
have
a
mailing
list
on
Google
Groups
and
you
can
hit
all
of
us
up
on
slack.
We
are,
you
know
available,
basically,
all
the
time
we
might
not
respond
right
away,
because
some
of
us
don't
check
it's
like
like
that
often,
but
it's
not
because
we're
ignoring
you
it's
just
whenever
we
check
slack
is
whenever
we
check
slack
I,
don't
use
slack
day
to
day
for
my
my
job,
so
it's
kind
of
intermittent
for
me,
but
yeah
I
will
respond
to
you
as
soon
as
I
see
it.
B
E
F
Thanks
for
the
talk,
so
my
question
is
around
instrumentation
libraries
right
so
sounds
like
you
moved
to
import
Elementary
for
tracing
and
I'm.
Maintaining
client
from
to
use
client
go
lines
I'm
interested
like.
What's
the
future
of
you
know,
metric
instrumentation?
Are
you
also
trying
to
kind
of
maybe
Converge
on
using
open,
Telemetry
go
for
metric
as
well
or
how
we
can
collaborate
like?
What's
the
future
here.
B
Oh
David
and
I
were
actually
just
talking
about
this,
so
we
did
a
lot
of
work.
Wrapping
the
premises.
Libraries
I
have
been
suggesting
to
David
that,
if
open
Telemetry
were
to
actually
natively
support
stability
classes
that
there
would
be
a
path
for
us
to
move
towards
towards
that,
otherwise
we
can
still
get
exemplars
through
the
Prometheus
Library,
so
yeah.
Those
are
the
options
on
the
table.
F
D
Yeah,
thank
you
and
I
would
say
as
well
like
we're
clearly
very
committed
to
metric
stability,
and
we
already
have
a
lot
of
Prometheus
endpoints,
so
whatever
we
do,
we
would
make
sure
that
we
maintain
you
know
the
same
metric
names
and
attributes
and
stuff.
So
in
some
ways
this
is
just
an
internal
implementation
detail
of
how
we,
you
know
how
we
manage
code
in
kubernetes.
A
D
I
think
operators
are
really
tricky
right
because
they
get
they're
driven
based
on
like
kubernetes
watch
events.
You
know
something
changed
in
kubernetes
and
now
they're
reacting
to
it.
So
we
haven't
quite
solved
the
context
propagation
puzzle
there,
but
it's
something
I
mean
I
gave
a
talk
in
2018
2019
on
a
proposal
that
unfortunately
hasn't
come
to
fruition.
So
it's
something
like
a
lot
of
people
are
interested
in
and
there
has
been
a
lot
of
excitement
historically
about
there's.
D
A
A
B
We
have
a
very
elaborate
static
analysis
framework
which
basically
runs
through
every
single
file
in
the
kubernetes
code
base,
analyzes
all
of
the
metric
definitions
and
then
outputs
a
yaml
file
with
basically
the
entire
metric
definitions.
And
then
we
parse
that
into
basically
a
markdown
file
which
ends
up
in
our
the
Kate's
website.
Directory.