►
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Intro: Kubernetes (Instrumentation) SIG - Frederic Branczyk, Red Hat
The intro to SIG Instrumentation is going to be a lightweight introduction to what sig-instrumentation is responsible for within Kubernetes and give guidance on how to contribute to SIG Instrumentation. Beyond that it will cover a high level overview of the work SIG instrumentation has done over the years and elaborate on current efforts including the sub projects being worked on.
https://sched.co/MPiA
A
Alright,
shall
we
get
started.
Ok,
thanks!
Everybody
for
coming
to
the
second
instrumentation
intro
I'm
Fredrik
I
work
at
Red
Hat.
You
can
find
me
on
Twitter
and
github,
and
today
I'm,
just
gonna,
give
you
a
quick
intro
to
what
sig
instrumentation
does
and
hopefully
that'll
encourage
you
to
join
our
meetings
and
participate
in
seconds
orientation.
A
So
let's
get
all
the
organizational
things
out
of
the
way
so
that
we
can
talk
about
technical
things,
so
myself
and
Piatra
who
I
do
not
see
in
the
audience
right
now,
but
I
do
think
he's
at
coop
con
we're
about
the
leads
for
cig
instrumentation.
We've
both
been
doing
it
I've
been
doing
it
for
a
little
bit
over
a
year
beatrice
since
SiC
instrumentation
was
founded
just
about
two
and
a
half
years
ago.
A
So
what
we
do
in
secure
instrumentation
is
we
define
and
we
kind
of
owned
the
best
practices
for
cluster
observer
ability
through
metrics
and
logging
in
kubernetes.
At
least
those
are
all
the
things
that
already
exist
in
kubernetes
today,
we'll
also
be
talking.
I'll
also
be
talking
about
some
of
the
things
that
we're
currently
working
on
and
want
to
introduce
into
kubernetes
that
haven't
happened
yet,
but
we're
actively
working
towards
so
I'll
go
through
all
of
these
things
and
I
left
a
good
amount
of
buffer
time
afterwards,
so
that
we
can
go
through
some
questions.
A
Hopefully,
you'll
have
some
at
the
end
of
this,
so
within
special
interest
groups
in
kubernetes,
I,
don't
know
if
you're
already
aware
of
this,
but
basically
when
you
create
a
special
interest
group,
you
define
what
the
scope
of
this
special
interest
group
is,
and
you
explicitly
say
these
are
the
things
that
we
want
to
work
on.
This
is
in
scope.
These
are
the
things
that
are
out
of
scope
and
force
against
maintainer.
We
really
we
don't
own
all
the
instrumentation
across
the
kubernetes
code
base.
A
That's
that
would
be
pretty
messy,
because
instrumentation
is
really
everybody's
responsibility
right,
everybody
should
be
adding
metrics
to
their
code.
Everybody
should
be
adding
meaningful
loglines
to
their
code
so
that
when
we
operate
our
kubernetes
clusters
that
we
can
actually
have
some
insight
into
what's
going
on
and
understand
and
troubleshoot
and
so
seconds
orientations,
responsibility
isn't
actually
to
go
ahead
and
do
all
of
that
stuff.
That's
as
I
said
responsibility
of
every
contributor,
but
we
define
kind
of
the
process.
A
A
A
Besides
that,
we
in
terms
of
projects
own
opt-in,
so
optional
components
for
kubernetes
clusters
and
anything
that
has
to
be
in
a
kubernetes
cluster
of
which,
actually,
we
realized,
there's
not
really
any
instrumentation
component.
That
is
totally
totally
unnecessary,
pretty
much
everything
is
upped
and
that
we
that
we
work
on
so
we've
talked
about
What's
in
scope.
What's
out
of
scope
is
even
though
I
personally
do
work
on
Prometheus
I,
as
this
as
one
of
the
sick
leads,
I
need
to
be
impartial
about
that
system.
A
Obviously,
Prometheus
has
been
chosen,
SP,
that's
the
expedition
format
that
wasn't
my
choosing,
but
I'm
still
happy
about
that.
But
in
general
the
kubernetes
SiC
instrumentation
is
not
about
how
these
signals
are
going
are
supposed
to
be
processed,
so
you
may
use
Prometheus,
but
there
are
other
systems
that
are
totally
compatible
with
the
Prometheus
format,
so
you
are
free
to
use
whatever
you
want
to
similar
with
logs.
A
You
can
ingest
the
logs
that
Prometheus
that
kubernetes
produce
produces
into
any
system
that
you
want
we're
not
making
any
any
assumption
about
this,
and
besides
the
logs
that
there
are
criminales
components
themselves,
produce
our
workloads
produce
logs
right
and
we
also
don't
dictate
what
kind
of
structure
or
anything
these
logs
have
to
be.
We
don't
even
own
the
the
act
of
writing
it
to
the
file.
A
So
now
that
we've
talked
about
on
a
high
level,
what
what
we
are
responsible
for
and
what
we
do.
Let's
talk
about
some
of
the
actual
sub
projects
that
we
that
we
work
on
so
probably
maybe
the
most
most
used,
one
in
terms
of
stars
by
from
our
sub
projects
as
coop
site
metrics.
If
you're
not
over.
Already
familiar
with
this
component,
it's
essentially
recently
people
have
been
saying
this
should
have
always
been
called
the
kubernetes
exporter.
A
You
can
actually
do
this
in
an
automated
matter,
so
this
is
basically
the
example
that
I
always
give
when,
when
I
talk
about
cube
state
metrics-
and
essentially
we
do
this
for
pretty
much
every
kubernetes
object
out
there.
This
is
a
really
really
useful
component.
If
you
haven't
checked
it
out,
I
highly
recommend
that
actually,
as
a
matter
of
fact,
because
you
because
premier
kubernetes
exposes
a
bunch
of
objects
right,
this
component
is
actually
a
little
bit
performance
sensitive.
A
So
large
kubernetes
clusters
can
easily
have
thousands
hundreds
of
thousands
of
objects
and
when
you
start
to
render
that
kind
of
text,
we've
actually
seen
megabytes
of
metrics
endpoints
output.
At
that
point,
depakote
becomes
performance,
sensitive
and
we've
done
actually
not
myself,
but
within
SiC
instrumentation
max
who's.
A
A
We
wanted
to
have
a
stable
API
within
kubernetes
that
could
be
used
by
auto
scaling
for
any
metric,
but
we
decided
to
divide
these
into
multiple
responsibilities,
and
so
we
created
something
the
big
picture
we
called
the
metrics
pipeline,
and
this
includes
resource
metrics,
custom,
metrics
and
external
metrics
resource
metrics
are
basically
things
that
all
your
workloads
have
in
common,
so
CPU
memory.
Potentially
there
are
more
they're,
just
not
part
of
this
API
yet,
but
potentially
that
this
could
be
extended
in
the
future.
A
The
kind
of
technical
aspect
of
this,
if
you're,
the
extension
API
is
in
communities
are
called
aggregated
API
is
the
way
that
works,
basically,
is
that
you
with
the
API
server
register
a
new
API
and
just
have
the
API,
the
actual
instance
of
the
API
running
in
your
kubernetes
cluster,
and
when
the
API
server
gets
the
request
for
this.
It
forwards
that,
to
that
extension,
API
process,
and
that's
exactly
the
way
that
we
define
this
now
there
can
be
a
lot
of
different
backends
to
this
right.
A
Pretty
much,
there's,
probably
a
vast
distribution
of
monitoring
systems
used
in
this
audience,
even
and
in
the
world,
it's
even
more.
So
what
we
did
here
is
actually
we
didn't
specify
you
have
to
use
Prometheus.
You
have
to
use
insert
whatever
monitoring
system
you're
thinking
of
right.
Now
we
only
defined
these
api's,
and
if
you
want
to
use
your
system,
then
there
has
to
be
a
shim,
an
adaptor
that
implements
this
API
for
your
or
your
particular
system,
and
just
because
resource
metrics
are
so
important.
A
We
actually
in
created
a
component
called
metric
server
which
implements
the
resource,
metrics,
API
and
basically
just
it's
a
very,
very
similar,
very,
very
minimal
system
that
just
scrapes
all
the
kubernetes
nodes,
the
some
what's
called
the
summary
API
and
gathers
all
the
metrics
from
those
nodes
stores
them
just
in
memory
to
only
the
latest
samples
and
when
you
then
use
coop
CTL
top,
for
example.
That's
where
coops
ETL
top
eventually
gets
its
data
through,
and
this
is
kind
of
the
we
at
some
point.
We
start.
A
We
said
that
this
is
probably
a
required
component
for
kubernetes
to
run
as,
but
eventually
we
realized.
Actually,
it
kind
of
makes
sense
that
even
this
API
and
it
is,
it
is
swappable
if
we're
already
having
let's
say
Prometheus,
for
example,
in
our
cluster
running.
Why
are
we
having
multiple
systems
grabbing
the
same
data
having
it
in
memory?
That's
kind
of
a
resource
waste,
so
this
is
this
kind
of
our
mindset
changed
a
little
bit,
and
we
said
this
is
probably
even
this
is
probably
a
optional
component.
A
As
I
said
all
this
is
in
memory.
It's
only
the
latest
sample,
that's
how
the
metrics
resource
metrics
API
is
defined,
and
this
is
not
actually
a
sig
instrumentation
project.
Yet
we've
talked
about
it
multiple
times,
but
there
are
a
couple
of
things
that
we
want
to
address.
I
happen,
or
my
team
at
Red
Hat
happens
to
work
on
this
component,
but
what
I
just
wanted
to
show
is.
All
of
this
is
totally
swappable
right.
A
This
happens
to
work
with
Prometheus,
but
there
is
a
stack
driver
adapter,
for
example,
and
it's
completely
completely
agnostic
and
last
but
not
least,
there
are
the
custom
and
external
metrics
api's.
These
are
again
just
general
definitions
of
api's
and
the
difference
is
that
custom,
metrics,
API
they're
always
associated
to
some
kubernetes
object.
A
So
you
could
have
the
queue
length
of
a
sum
deployment,
but
in
external
metrics
you
only
have
a
namespace
separating
this,
so
this
is
kind
of
used
to
Bill
to
integrate
into
the
tenancy
model
of
your
back-end
monitoring
system
so
that
you
can
have
some
amount
of
separation.
But
you
need
to
be
aware
that
probably
whoever
is
using
them
external
metrics
API,
probably
has
a
little
bit
more
privilege
or
larger
scope
of
access
to
metrics.
A
Then
if
you
were
to
use
the
custom
metrics
API
now,
as
I
said,
because
of
the
swappable
violate
mileage
may
vary
across
adapters,
but
try
out
the
ones
for
your
the
monitoring
system
that
you're
using
and
I
think
pretty
much
all
the
implementations
are
open-source,
so
just
give
people
feedback.
So
that's
pretty
much
all
the
things
that
we've
built
so
far
as
sig
instrumentation
and
that
we're
still
maintaining
right
now.
We,
there
is
actually
one
component
that
I
haven't
mentioned,
that
we
used
to
maintain
called
hipster.
A
We
deprecated
this
component
a
while
ago,
I
think
I
actually
first
mentioned
this
at
open
source
summit
in
Prague,
end
of
2017,
I,
think
I,
don't
think
I've
mentioned
heaps
or
much
since
then
so
I'm,
hoping
that
maybe
maybe
in
this
room,
people
don't
even
know
hipster.
Then
I've
done
my
job
so
but
anyways.
A
Unfortunately,
we
weren't
always
involved
in
the
review
process,
which
may
or
not
be
good
anyways.
So
what
happened
is
that
over
time
a
number
of
metrics
were
introduced
into
kubernetes
that
actually
violated
those
guidelines
and
essentially
we
went
through
and
for
the
kubernetes
1.14
release.
We
hopefully
caught
all
the
cases
that
violated
these
guidelines
and
deprecated
the
old
metrics
introduced
a
new
metric
that
has
does
not
violate
the
guidelines
anymore.
The
old
metrics,
the
deprecated
ones,
are
still
available
in
kubernetes
1.14.
They
will
be
removed
in
1.15.
A
So
one
point:
fourteen
is
kind
of
the
release
where
you
want
to
switch
actually
part
of
the
kubernetes
enhancement
proposal
was
kind
of
there's,
always
a
risk
mitigation
section.
So
one
of
the
things
that
we
did
is
we
looked
at
the
community
and
identified
some
of
the
projects
where
they're
the
biggest
sources
of
dashboards,
biggest
sources
of
alerts,
and
we
said
to
minimize
the
risk
for
our
users
and
to
minimize
the
pain
of
migration.
We'll
update
some
of
these
sources
so
that
users
just
need
to
update
those
and
they'll
be
good
to
go
again.
A
Unfortunately,
we
can
do
this
for
absolutely
everyone,
especially
for
cloud
providers
that
aren't
for
where
some
of
these
things
are
not
necessarily
public.
We
couldn't
we,
we
just
communicated
this
a
lot.
We've
announced
it
at
the
community
meeting
at
various
communities,
SiC
instrumentation
meetings-
and
so
hopefully
everybody
by
now
has
realized
that
this
is
happening
and
that
increment
is
1.4
15.
These
metrics
will
be
removed.
A
So
yes,
so
all
the
corrected
things
have
landed
in
fort
1.14
and
deprecated
ones
will
be
removed
very
soon
and
now
I'll
have
Han
come
up
because
I
feel
like
this
is
a
really
important
topic
and
haunt
since
Han
happens
to
be
here
and
works
on
this
topic.
I
felt
like
it's
his
place
to
talk
about
this
not
me.
B
B
So
what
we
would
like
to
do
is
we
would
like
to
start
treating
metrics
properly
as
an
API.
This
means
that
metrics
will
have
to
undergo.
Metrics
will
have
various
stages
like
alpha
metrics,
possibly
beta
metrics
and
staple
metrics,
and
they
will
officially
go
through
a
deprecation
policy
in
order
to
get
removed.
This
means
that
if
you
have
monitoring
or
alerting
based
off
of
these
metrics,
then
you
can
expect
a
certain
lifecycle
of
stability.
So,
specifically,
the
way
that
this
works
is
once
you
have
a
stable
metric.
B
If
you
would
like
to
deprecate
that
metric,
if
a
metric
does
in
fact
have
become
deprecated,
then
it
will
for
the
next
kubernetes
release.
It
will
be
annotated
as
a
dedicated
metric
and
a
subsequent
release.
It
will
not
appear
in
a
metrics
endpoint.
This
could
be
the
API
server
or
binary.
This
could
be
a
cubelet
binary.
This
could
be
controller
manager,
but
you
will
be
able
to
optionally
re-enable
this
during
the
deprecation.
B
So
if
you
notice
that
an
alert
is
not
working
or
a
you
know,
graph
isn't
displaying
data,
then
you
can
rien
a
ballista
will
give
you
time
due
to
migrate
off
of
the
the
deprecated
metric,
so
what's
coming
up.
Next
is
basically,
we
are
in
the
middle
of
working
out
static
analysis
for
validating
metrics,
so
if
any
of
you
guys
are
familiar
with
the
way
that
kubernetes
open
source
does
conformance
testing,
it's
very
similar
run
stack
analysis
against
the
code
to
see
whether
or
not
you
know
conformance
tests
have
been
properly
approved.
B
And
you
know
one
of
the
things
about
sig
instrumentation
is
that,
while
we
there
are
code
bases
that
sick
instrumentation
does
own
in
the
kubernetes
base
the
main
code
base,
there
are
many
other
things
which,
on
the
different
binaries
and
you
know
so
far,
it
has
been
very
difficult
for
sig
instrumentation
to
be
sort
of
involved
on
the
level
of
a
PR
right.
So
like
somebody
from
API
machinery
or
somebody
from
you
know,
sig
note
could
be
introducing
some
features
and
metrics,
and
you
know,
while
their.
A
B
Is
good
they
might
not
understand.
You
know
the
guidelines
which
are
prescribed
by
sig
instrumentation,
so
basically
static
analysis
tool
and
will
allow
us
to
determine
when
metrics
have
been
introduced
so
that
sig
instrumentation
can
be
more
properly
involved
in
in
reviewing
metrics
and
ensuring
that
you
know
the
stuff
for
the
metrics
overhaul
continues
to
be
there
in
the
future.
A
Okay,
so
that's
probably
the
most
most
discussed
topic
right
now
in
our
in
our
meetings.
So
if
you're
interested
in
that
like
join
our
next
one,
which
is
next
Thursday
but
I'll
put
that
up
later
later
on
the
slide,
but
there
are
while
this
is
something
that
we're
concretely
working
on
already
or
primarily
Han.
Actually
so
thanks
Han
for
doing
all
this
great
work,
there
are
a
couple
of
other
discussions
that
are
going
on
there.
A
There
are
kind
of
a
little
bit
more
loose,
but
I
think
they're,
they're,
exciting
discussions
to
have
and
I
think
mostly
where
we've
already
said.
All
of
these
things
are
great
to
have,
but
we
need
to
discuss
them
a
little
bit
more.
We've
just
started
discussing
these
things,
so
maybe
out
of
those
the
most
concrete
one
that
we've
discussed
the
most
is
watch
support
in
the
metrics
API.
So
as
of
right
now,
the
metrics
API
is
only
support
list
and
get
verbs.
A
So
essentially
you
can
only
list
all
the
metrics
and
you
can
or
you
can
get
a
single
metric.
But
if
you
you're,
probably
familiar
with
the
kubernetes
api,
it's
work,
you
can
usually
do
coop
CTL
w2
watch
and
you
get
all
the
notifications
or
all
the
changes
that
happen
to
this
to
this
particular
object,
and
the
reason
why
this
could
be
useful
for
metrics
is
that
from,
for
example,
in
like
a
scale
to
zero
kind
of
use
case
where
we
want
to
scale
up
pretty
fast.
A
It's
really
important
that
we
get
these
metrics
really
fast
and
react
to
them
really
fast.
So
a
watch
API
could
be
desirable,
but
actually
in
general,
if
I
it's
been
two
and
a
half
years.
Actually,
since
we
introduced
the
metrics
api's
I
believe
back,
then,
when
we
talked
to
sick
API
machinery,
we
were
told
for
this
API
to
ever
actually
graduate
to
a
stable
API.
This
must
have
a
what
must
must
have
watch
capabilities,
I
believe
we
should
probably
go
back
and
reiterate
on
this.
This
discussion
point
but
I
believe
that's.
A
That
was
that
was
kind
of
the
message,
but
in
general
like
this
is
a
difficult
topic,
because
essentially
we
did
a
little
bit
of
research
and
essentially
no
monitoring
system
out
there.
We
know
actually
supports
an
API
to
implement
this
right
now,
but
potentially,
if
we
do
this
and
we
implement
it
in
metric
server,
it
should
be
fairly
part.
It
should
be
possible
to
do
this
in
metric
server.
Maybe
we
can
incentivize
to
other
monitoring
systems
to
build
api's
like
this
yeah,
so
I
think
this
is
an
exciting
topic.
A
I'm
excited
to
talk
about
this
more
but
yeah,
it's
it's
challenging.
So
if
you're
interested
in
these
kind
of
things
come
join
us,
then
one
of
the
other
things
that
we're
discussing
currently
and
actually
yesterday
we
had
our
face-to-face
meeting
at
the
kubernetes
contributor
summit,
and
this
is
one
of
the
topics
that
we
discussed
quite
a
lot
there.
How
can
we
introduce
structured
logging
into
kubernetes
components
if
you're
familiar
right?
Now?
A
It's
basically
an
unstructured
log
line
using
the
blog
or
now
now
we
actually
fork
the
glog
library
fund
from
Google
into
Kellogg,
so
that
we
could
could
fix
a
couple
of
immediate,
immediate
issues
that
we
had
with
it.
But
we
want
to
go
further
and
we
want
to
have
real
structured
logs
so
that
we
can
better
do
better
more
structured
analysis
on
these
kinds
of
things.
But
it's
difficult
introducing
this
kind
of
a
change
because
some
people
are
are
already
are
relying
on
the
struck
on
the
text
that
is
already
being
logged
by
kubernetes.
A
So
this
needs
a
little
bit
more
thought
and
these
discussions
are
already
ongoing
and
then
last
not
but
not
least-
and
this
is
basically
what
I
was
referring
to
on
the
very
first
slide.
We
are
already
responsible
for
metrics
and
logging
and
we
would
like
to
do
also
tracing,
but
we
haven't
actually
gotten
to
that.
Just
yet.
But
we're
super
excited
about
this
space.
A
So
with
that,
if
you
are
interested
in
any
of
what
I
just
said,
you're
more
than
welcome
to
join
our
bi-weekly
meetings,
which
are
at
5
p.m.
UTC
but
their
bi-weekly,
so
the
next
one
is
actually
next
Thursday
next
week
Thursday
and
then
every
two
weeks,
and
if
you're
you
can
just
take
a
picture
or
I'll
put
the
slides
up
on
schedule
as
well.
You
can
just
dive
into
the
meeting
notes.
It's
totally
free
form.
A
A
Yeah
so
so
the
question
was
as
we're:
deprecating
a
bunch
of
metrics
for
we
deprecated
them
for
one
point,
one
point:
fourteen
and
they're
removing
them
for
one
point:
fifteen:
what
is
the
I
guess?
What
is
the
exact
risk
mitigation
strategy
that
we
went
with
so
there
is
a
project
called
the
kubernetes
mixin,
which
is
a
project
that
is
a
set
of
alerting
rules
and
Gravano
dashboards
written
in
JSON
it,
and
that's-
we've
identified
that
as
probably
the
largest
use
of
learning
rules
and
dashboards
in
within
the
community.
A
If
there's
anything
else
that
you
know
about,
please
tell
us,
but
we
do
know
that
this
is
a
really
really
popular
project.
So
this
is
what
we
already
updated.
So
there
is
a
release
status.
We
actually
did
release
just
about
two
weeks
ago.
That
is
the
last
release
that
supports
communities
1.13
and
from
now
on,
basically
on
from
from
master
and
every
fort
going
release
from
that
project
will
have
kubernetes,
1.14
plus
support,
so
with
all
the
new
metrics
that
we've
introduced
and
that
will
hopefully
become
stable.
A
Okay,
so
the
question
was:
how
how
can
you
participate
and
contribute
so
I
think
one
of
the
best
ways
is
to
participate
in
the
meetings
and
essentially
hear
about
what's
what's
currently
ongoing,
and
usually
the
topics
are
large
enough
for
more
than
it's
way
more
than
a
single
person
to
handle.
So
it's
very
easy
to
get
involved
through
that
other
than
that
kubernetes
issues
and
pull
requests
and
everything
is,
are
labeled
with
special
interest
group
specific
labels.
A
So
if
you're
interested
in
SiC
instrumentation,
then
just
filter
all
the
issues
and
PRS
by
by
the
second
fermentation
label-
and
you
can
just
go
ahead-
we
like
having
a
community
start
doing
reviews
that
as
a
maintainer
of
a
project
that
is
like
the
that
makes
me
the
happiest,
because
if
the
community
does
a
does
a
review
that
I
don't
have
to
do
and
it's
high-quality
then
that's
just
that
makes
my
day.
So
please
do
any
other
questions.