►
From YouTube: SIG Instrumentation Introduction and D... Han Kang, David Ashpole, Elana Hashman, Frederic Branczyk
Description
Don’t miss out! Join us at our next event: KubeCon + CloudNativeCon Europe 2022 in Valencia, Spain from May 17-20. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
SIG Instrumentation Introduction and Deep Dive - Han Kang & David Ashpole, Google; Elana Hashman, Red Hat; Frederic Branczyk, Polar Signals
Kubernetes SIG Instrumentation is responsible for ensuring high quality and consistent instrumentation across the Kubernetes project. We will begin with an introductory overview of the efforts the SIG Instrumentation has worked on in the past and is currently working on. This deep dive session will go in detail currently ongoing efforts happening within SIG Instrumentation to share with the audience concrete pieces of work to encourage future collaboration. Software engineering and operations are both disciplines practiced in SIG Instrumentation, and any experience will help the special interest group's mission. Join this session to learn how to get involved in SIG Instrumentation to make instrumentation even better!
A
All
right
welcome
everyone.
This
is
sick,
instrumentation,
intro
and
deep
dive.
I'm
frederick
I'm
the
founder
of
polar
signals,
and
I
am
one
of
the
tech
leads
for
the
special
interest
group
for
instrumentation
and
kubernetes,
and
today
with
me,
I
have
david
ashbill
who's.
Also,
a
tech
lead
and
he
works
at
google
he's
a
deck
lead
at
within
sick
instrumentation
as
well.
We've
also
got
elana,
who
is
one
of
the
chairs
of
sick
instrumentation.
She
works
at
red
hat
and
we've
got
han,
who
also
works
at
google
who's,
also
a
chair.
A
So
if
you
don't
already
know
special
interest
groups
in
kubernetes,
we
have
several
of
these,
and
essentially
there
are
focus
groups
for
certain
areas
to
focus
on
and
we
focus
on
instrumentation
and
we'll
talk
a
little
bit
more
in
depth.
What
we
actually
mean
by
that,
but
essentially
we
take
care
of
all
things,
observability
right
and
today,
let's
I
want
to
talk
about
a
little
bit
like
about
that
definition
and
a
little
bit
about
how
we
work
as
a
special
interest
group.
A
Then
I'm
going
to
walk
over
a
couple
of
our
sub
projects,
then
elena's
going
to
talk
about
logging
and
some
of
our
initiatives
within
logging,
then
david
is
going
to
talk
about
tracing
and
han,
is
going
to
finish
up
with
metrics
and
then
we'll.
Let
you
know
a
little
bit
about
how
you
can
contribute
where
you
can
find
us
and
some
related
talks
to
these
topics.
A
So
essentially
the
way
all
instrumentation
or
all
special
interest
groups
in
kubernetes
get
created
is
they
have
a
charter?
They
have
a
specific
purpose,
and
this
is
an
a
literal
excerpt
from
our
charter,
our
sick,
instrumentation
charter,
and
our
purpose
is
to
cover
the
best
practices
for
cluster
observability
across
all
components
and
develop
relevant
components
so
effectively.
What
this
means
is
that
you
know
we
care
about
more
than
just
the
kubernetes
kubernetes
repository,
although
we
care
a
lot
about
this
obviously,
but
we
also
create
additional
components.
A
B
A
We
are
not
necessarily
not
for
all
metrics.
Are
we
a
blocking
review,
but
for
some
we
may
be,
especially
for
our
stability
gate
guidelines,
so
the
more
stable
a
metric
becomes
in
terms
of
stability
guarantees,
the
stricter
the
reviews
become
as
well,
and
this
is
again
a
more
general
pro
process
within
kubernetes
whenever
we
develop
larger
scale,
things
that
are
more
involved,
we
create
these
things
called
kubernetes
enhancement
proposals
or
in
the
kubernetes
community,
typically
they're
just
referred
to
as
caps.
A
A
I
picked
these
three
ones
because
they're
the
ones
that
I'm
most
familiar
with
and
that
I
think
the
the
kind
of
most
important
ones
that
we
take
care
of
so
kubesave
metrics
is
the
one
that
I
personally
have
other
than
the
kubernetes
repository
been
involved
with
the
longest
and
probably
the
most,
even
and
essentially
what
group
state
metrics
is.
If
you're
familiar
with
the
prometheus
ecosystem,
it's
a
you
can
think
of
it
as
a
prometheus
exporter
for
a
kubernetes
cluster.
A
Essentially,
what
we
do
is
we
look
at
the
kubernetes
api
and
anything
that
could
be
possibly
useful
as
a
metric
in
prometheus.
We
kind
of
convert
to
a
prometheus
style
metric,
and
so
this
way
we
get
information
about
parts
about
deployments
about
stateful
sets
about
all
of
these
wonderful
things
so
that
we
can,
then
you
know,
create
alerts
or
dashboards
out
of
to
you
know,
visualize
and
make
useful,
and
just
just
a
really
quick
example
that
I
wanted
to
bring
you
to
to
this,
to
highlight
the
usefulness
of
this
component.
A
This
is
an
actual
example
of
what
we
have.
We
have
the
expected
replicas
of
a
deployment
and
we
have
the
actual
replicas
of
a
deployment
and
why
this
is
useful
is
now
we
can
kind
of
compare
these
two
numbers
and
we
can
understand
whether
a
deployment
has
been
rolled
out
successfully
right-
and
this
is
just
one
of
many
useful
examples
that
you
could
use
this
for
and
kubesate
metrics
exposes
a
lot
of
metrics
and
it's
highly
optimized.
A
So
whenever
there's
a
call
to
the
kubernetes
api
about
a
metrics
server,
a
metrics
api,
the
kubernetes
api
just
forwards
this
to
the
metric
server
and
the
metric
server.
Essentially
asynchronously
collects
these
metrics
from
kubernetes
nodes
and
then
returns
them
on
requests
and
in
a
way
the
prometheus
adapter
is
actually
the
exact
same
thing,
except
that
it
doesn't
re-implement
all
of
this
gathering
functionality.
A
The
really
cool
thing
about
the
prometheus
adapter
also
is
it
doesn't
only
speak
the
resource
metrics
api,
but
it
also
speaks
the
custom
and
external
matrix
apis,
and
this
is
really
cool,
because
now
we
can
not
only
auto
scale
based
on
cpu
and
memory
usage,
but
we
can
auto
scale
in
any
metric
that
we
have
in
our
prometheus
server.
This
is
a
really
really
powerful
tool
to
have,
and
we
actually
just
kind
of
migrated
this
project
previously
this
was
maintained
by
sully.
A
Thank
you
sully
for
having
maintained
this
project
for
such
a
long
time
and
now.
Finally,
it's
been
it's
now
under
the
umbrella
of
the
sick
instrumentation.
I
believe
we
just
a
couple
months
ago
finished
this
migration,
so
yeah.
These
are
the
kind
of
three
sub-projects
that
I
wanted
to
highlight
to
you
today
and
now
elena
goes
on
with
logs.
C
So
much
of
our
efforts
have
been
dedicated
towards
transitioning
kubernetes
to
structured
logs.
Now
you
may
ask
what
is
a
structured
log,
so
I've
demonstrated
on
the
slide
as
an
introduction
here
is
sort
of
what
the
before
and
after
looks
like.
So
the
before
is
the
first
log
line,
and
that
was
what
a
log
line
in
the
cube
looked
like
prior
to
transitioning
to
structured
logging
as
well
as
including
these
after
views.
C
Now,
I've
included
both
text
and
json
versions
of
the
log,
because
by
default
the
cubelet
will
continue
to
log
in
text
mode,
but
you
can
also
turn
on
json
mode,
which
then
can
be
ingested
by
various
log
aggregation
tools
and
make
indexing
of
these
log
entries
much
easier
so
which
brings
us
benefits
of
structured
logs.
Why
would
we
do
this?
Why
would
we
want
to
do
this?
It
makes
it.
C
To
aggregate
and
correlate
logs
by
presenting
them
in
a
fashion,
we
don't
have
to
parse
them
all
after
the
fact
they're
already
in
a
serializable
format,
and
then
various
tools
can
deal
with
them
rather
than
dealing
with
just
the
raw
syslog
or
we
might
have
to
use
regex's
or
other
things
to
parse
those
log
entries
in
order
to
determine
what
happened
where
so
structured
logs.
When
so,
we
started
out
by
migrating
the
cubelet
in
the
121
release
and
even
sort
of
track.
C
Our
progress
in
the
attached
issue
and
part
of
that
work
was
including
some
static
analysis
to
prevent
regressions.
So
every
file
that
was
migrated
in
the
cubelet
was
marked
such
that
ci
will
ensure
that
if
you
go
and
add
a
non-structured
log,
it
won't
pass.
You
have
to
go
and
ensure
the
logging
entry
is
structured.
C
Now
these
flags
have
been
supported,
since
we
initially
used,
I
guess,
g-log
and
then
later
transitioned
to
k-log,
but
these
flags
kind
of
came
along
as
we
were
implementing
log
with
g
log
support
they
weren't
necessarily
intended
to
be
part
of
kubernetes.
We
just
happened
to
get
this
feature
set
and
now,
when
we're
trying
to
implement
flag
parity
between
text
logging
and
json
logging
we're
finding
that
it
is
relatively
difficult
in
order
to
support
this
quite.
D
C
D
C
An
enhancement
in
progress
targeted
for
the
123
release
that
will
deprecate
a
number
of
these
flags
using
the
standard,
kubernetes
deprecation
process
and
remove
them
from
kubernetes
components.
So
not
in
scope
here
is
removing
them
from
k.
Log
itself
they'll
stay
in
the
library.
We
just
won't
support
them
in
kubernetes
anymore.
C
So
who's
working
on
structured
logging,
one
of
the
more
exciting
things
that
I
get
to
talk
about
is
the
creation
of
our
new
working
group
structured
logging.
In
order
to
manage
this
structure
block
migration,
so
the
organizers
of
the
new
working
group
are
merrick
for
google
and
shubhaksha
from
apple
and
I'm
very
excited
that
they've
stepped
up
to
lead
this
effort.
They
have
a
slack
channel,
so
you
can
check
it
out
there.
They
also
have
charter
and
the
community
repo
and
they
meet
what
bi-weekly
on
thursdays
at
1500
utc.
C
So
they
need
your
help.
I
know
that
as
part
of
the
cubelet
migration
in
121,
we
had
a
lot
of
new
contributors
who
helped
out
with
that
effort,
so
I'm
sure
that
there
will
be
lots
to
do
in
the
123
release
and
onwards.
So
it's
definitely
a
great
place
to
get
involved.
If
you're
looking
for
new
things
to
do
in
kubernetes,.
C
D
B
Thanks
elena,
a
lot
has
happened
since
the
last
kubecon
in
the
area
of
distributed
tracing.
My
name
is
david
ashpole
and
I've
had
the
privilege
of
working
in
this
area
and
I'm
excited
to
share
the
progress
we've
made,
control
plane,
tracing,
which
includes
both
the
api
server
and
ncd,
is
now
an
alpha.
B
This
is
something
that
the
sig
has
been
working
towards
for
a
very
long
time
and
it's
really
exciting.
To
finally
see
it
happen.
A
special
shout
out
to
lily
for
her
work
on
the
xcd
integration
control,
plane
tracing,
allows
cluster
operators
to
collect,
distributed
traces
for
correct
requests
sent
to
the
kubernetes
control
plane.
This
will
make
it
easier
to
debug
slow
requests
and
to
figure
out
what
path
the
request
took
through
the
system.
B
B
B
I
want
to
dive
a
little
bit
deeper
into
control,
plane
tracing
since
it
just
reached
alpha.
So
what
I
did
for
this
demo
is,
I
enabled
the
api
server
tracing
feature
gate
and
on
the
api
server.
I
set
the
tracing
config
file
flag
to
a
file
containing
the
following
configuration,
but
sets
a
one
percent
sampling
rate
for
the
api
server.
B
So
here's
an
example
trace
you
can
see.
The
top
bar
is
the
incoming
request.
So
the
request
first
hit
the
api
server
at
the
very
beginning,
and
then
the
api
server
responded
at
the
very
end,
and
so
we
can
see
the
teal
lines
are
coming,
our
spans
that
are
emitted
by
the
api
server.
So
that
includes
this
top
one
and
both
of
these
child
spans
here
and
then
I'm
also
running
a
custom
mutating
admission
controller
here
and
fcd
has
its
tracing
enabled
as
well,
and
that
span
is
here
so
from
this
example.
B
We
could
see
all
the
way
from
when
the
api
server
first
got
the
request
to
when
it
asked
for
a
response
from
a
mutant
mission
controller
to
when
it
actually
committed
the
transaction
to
etcd
and
responded
to
the
user.
So
this,
hopefully,
will
be
really
useful
to
operators.
As
I
said-
and
this
is
just
the
beginning-
we
know
that
tracing
becomes
more
useful.
The
more
things
integrate
with
it.
So
we're
excited
to
see
where
this
all
leads
up
next
to
talk
about,
metrics
is
han.
Take
it
away.
D
D
This
means
that
the
primary
kubernetes
components
the
cube
api
server,
the
controller
manager
scheduler
cubelet.
All
of
these
things
expose
text
based
metrics
endpoints,
which
can
then
be
scraped
and
ingested
by
a
various
different
number
of
time.
Series
back
ends,
though
commonly
this
tends
to
be
prometheus
prometheus,
has
a
number
of
different
metric
types
which
I
won't
get
into
since
this
isn't
a
prettiest
talk,
but,
needless
to
say,
we
use
basically
all
of
them
in
kubernetes.
D
There
is,
though,
usually
a
common
theme
underlying
these,
which
is
that
metrics
can
grow
in
size
and
quickly
become
memory
leaks.
This
can
cause
instability
in
the
component.
Why
is
this
the
case?
Let's
take
a
look.
Take
this
example
metric.
It's
a
request
metric
like
all
metrics.
It
has
a
width
and
a
height
by
width.
What
I
mean
is
the
number
of
labels.
D
D
D
D
And
we
did
that
we
created
a
framework
which
makes
the
structure
of
a
metric
an
immutable
api.
We
call
this
the
metric
stability
framework
with
static
analysis
and
hooks
into
kubernetes
commit
pipelines.
We
can
validate
that
stable
metrics
do
not
structurally
change;
they
will
always
have
the
same
labels.
You
will
cannot
remove
or
you
cannot
add
labels
of
a
stable
metric.
D
D
D
D
That
means,
let's
say
upstream
kubernetes,
makes
mistakes
and
releases
a
bad
metric,
one
that
explodes
in
cardinality,
because
a
thousand
new
labels
have
been
added,
you
don't
have
to
roll
back
your
cluster.
All
you
have
to
do
is
restart
it
with
a
flag
and
bind
the
possible
number
of
labels
for
that
metric.
D
D
One
of
the
ways
that
we
want
to
do
this
is
to
extend
metric
stability.
We
want
to
bring
it
into
parity
with
feature
stages.
Currently
we
actually
only
have
two
stable
classes
alpha
and
stable,
and
the
former
has
absolutely
no
guarantees
and
stable
metrics
have
very
firm
guarantees.
We
want
additional
expressiveness
to
denote
metrics,
which
are
kind
of
in
the
middle,
and
you
might
be
able
to
set
up
charts,
for
we
also
want
to
hook
into
the
static
analysis
pipeline
to
auto,
generate
metric
documentation.