►
From YouTube: Kubernetes SIG Instrumentation - Intro - Frederic Branczyk, Red Hat & Piotr Szczesniak, Google
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Kubernetes SIG Instrumentation - Intro - Frederic Branczyk, Red Hat & Piotr Szczesniak, Google
Kubernetes SIG Instrumentation is responsible for ensuring high quality and consistent instrumentation across the Kubernetes project. This intro session will give an overview of the efforts the SIG Instrumentation has worked on in the past and is currently working on. Software engineering and operations are both disciplines practiced in SIG Instrumentation, and any experience will help the special interest group's mission. Join this session to learn how to get involved in SIG Instrumentation to make instrumentation even better!
https://sched.co/Uaix
A
Some
over
all
things,
hopefully,
everyone's
already
familiar
with
what
special
interest
groups
within
communities
are,
if
not
I'll,
expand
it
real,
quick,
so
essentially,
special
interest
groups
within
kubernetes
are
just
like
actually
formally
defined
groups
around
like
cloud
providers
or
instrumentation
or
API
machinery,
so
like
people
who
commonly
work
together,
basically
formed
groups
within
kubernetes
to
more
efficiently
work
together
and
instrumentation,
as
the
name
implies,
is
all
about
like
observability
and
those
kinds
of
things,
but
we'll
dive
into
the
formalities.
A
little
bit
more
but
like
to
start
out
with
Katherine
and
I.
A
So
I
think
you've
been
you've
been
a
lead
ever
since
we
found
it
especially
like
the
instrumentation
I
was
there
when
we
founded
it,
but
Fabien
was
the
the
lead
back
then,
and
then
I
took
over
from
him
roughly
a
year
and
a
half
ago.
So
yeah,
that's
kind
of
the
organizational
thing
and
like
every
special
interest
group
within
within
kubernetes,
has
a
mission
statement
so
right
now.
This
is
when
look
at
the
special
interest
group
charter.
A
Basically,
every
special
interest
group
has
a
written
down
document
of
what
their
mission
is
and
what
work
is
in
scope
for
them.
What's
out
of
scope
for
them,
so
that
everybody
can
understand
what
each
of
these
groups
do
and
our
mission
is.
We
define
best
practices
for
cluster
observability
through
metrics
and
logging
across
all
components
within
communities,
and
even
even
though,
when
we
say
when
we
tell
you
it's
a
crew,
all
of
kubernetes,
that's
kind
of
vague
right.
A
We
roughly
mean
the
kubernetes
community's
project
on
github,
but
because
a
lot
of
these
things
are
also
ripped
out
into
libraries
and
stuff,
like
that,
we
happen
to
also
influence
other
project
through
that
now
you
probably
are
already
wondering
what
we're
seeing.
We
see,
metrics
and
large
sum
here,
what
about
traces,
we'll
dive
into
that
a
little
bit
later
or
touch
on
it,
and
then
there's
also
the
deep
dive
tomorrow,
where
you'll
hear
some
of
this
ongoing
work,
so
I
highly
recommend
you
going
to
that
tomorrow.
A
So,
for
example,
one
of
the
very
first
things
that
we
did
is
with
sig
instrumentation
is
we
wrote
down
best
practices
about
the
metrics,
intrument
instrumentation,
and
things
like
that,
so
that
when
someone
uses
metrics
exposed
by
kubernetes,
they
actually
have
a
comprehensive
and
a
consistent
experience,
and
while
there
are
a
number
of
violations
to
those
best
practices
today
we're
trying
to
tackle
that
and
dive
into
that
initiative
in
a
bit.
But
overall,
the
idea
is
that
we
want
kubernetes
to
have
really
high
quality
metrics
and
that's
why
we
need
these.
A
Those
kinds
of
guidelines
and
frameworks,
and
more
generally
sig
instrumentation,
is
a
super
cross,
sig
kind
of
special
interest
group
API
machinery,
for
example.
Obviously
it
has
a
very,
very
defined
space
that
they
work
in
right,
whereas
instrumentation
is
like
spread
out
through
the
entire
code
base.
So,
if
you're
looking
to
get
to
really
touch
every
single
component
within
kubernetes
seconds
orientation
is
a
great
place
to
be.
It
can
be
scary,
but
folks
are
generally
nice
and
in
kubernetes,
so
I
highly
recommend
it
yeah
and
so
other
than
that.
A
Obviously
we
expose
metrics
in
Prometheus
format,
for
example,
but
that
format
is
actually
compatible
with
a
bunch
of
monitoring
solutions
out
there.
So
we
don't
we
don't
say
you
have
to
use
Prometheus.
You
can
use
any
other
system
that
can
ingest
the
Prometheus
format,
for
example
same
with
logs
same
with
anything
else
that
we
would
expose
and
specifically
kind
of
going
along
those
lines.
We
don't
own
anything
specific
to
any
cloud
providers.
All
those
things
belong
in
the
SIG's
specific
to
each
of
those
cloud
right
or
so.
A
There's
a
special
interest
group
for
for
Google
cloud,
for
example,
there's
one
for
sure,
there's
one
for
AWS.
So,
if
there's
anything
that
is
very
specific
to
instrumentation
or
monitoring
or
anything
on
a
specific
cloud
provider,
then
that
belongs
in
that
special
interest
group
and
kind
of
our
in
the
same
way
as
we
don't
dictate
which
system
you
have
to
use
for
metrics
or
for
partnering.
We
also
do
the
same
with
logs
and
basically,
the
only
interface
that
we
have
for
logging
systems
is.
A
A
So
one
of
the
concrete
components.
This
is
an
add-on
component,
so
you
don't
necessarily
have
to
run
this,
but
this
is
one
other
component
that
we,
for
example
own.
This
is
cube
state
matrix.
Essentially,
what
coupe
state
matrix
does
is
it
watches
the
kubernetes
api
and
it
generates
metrics
off
of
all
of
those
objects.
So
one
very
common
example
is
what
I
have
on
on
the
slides
here.
I
think
the
the
benefit
of
this
is
kind
of
obvious.
A
We
exposed
metrics
about
the
expected
amount
of
replicas
about
of
a
deployment
and
the
actual
amount
of
replicas,
and
so,
if
there's
a
mismatch,
for
example,
you
can
alert
on
that
kind
of
state
and
basically
everything
that
can
be
somehow
squeezed
into
the
form
that
looks
like
a
metric.
We
do
in
coop
site
metrics,
so
basically,
every
API
object
that
you've
ever
seen
probably
has
metrics
and
cube
state
metrics
to
do
useful,
alerting
with
coop
site
metrics
has
existed
for
a
really
long
time,
I
think
it
was
actually.
A
It
was
probably
one
of
the
very
first
things.
I
thought
that
I
ever
contributed
to
within
kubernetes.
So
I
think
this
is
a
really
cool
project
to
get
involved,
because
there's
almost
certainly
some
metric
that
we're
missing
and
if
you
have
a
use
for
it,
if
you
want
to
graph
it,
if
you
want
to
alert
on
it,
we're
super
happy
to
accept
those
kinds
of
metrics
and
so
yeah,
but
over
the
over
the
years.
A
We
are
also
accumulated
a
couple
of
mistakes
that
we've
made
some
technical
depth,
and
so
that's
why
we're
also
planning
version
two
of
coop
state,
metrics,
crib,
set
metrics
I,
think
we
called
it
1.0
roughly
two
years
ago,
maybe
two
and
a
half
years,
even
so,
yeah
we're.
This
is
also
a
really
cool
way
to
get
involved.
There
is
actually
an
umbrella
issue
where
we
track
all
the
work
that
we
want
to
do
to
to
move,
to
move
forward
to
to
next
major
version
of
this
yeah.
A
A
And
it's
really
a
it's
really.
It's
a
really
simple
component.
Basically,
what
it
does
is
it
fans
out
to
all
the
nodes
in
your
cluster,
hits
a
specific
specific
API
on
the
couplet
and
then
keeps
only
the
very
latest
samples
for
each
container
in
your
infrastructure
in
memory
and
whenever
you
do
coups
ETL
top,
it
returns
exactly
those
that
you've
requested.
A
Everybody
could
rely
on,
but
have
this
generalized
API,
so
that,
if
you
already
have
happen
to
have
a
monitoring
system
that
is
it
collecting
all
of
this
data
that
would
be
doing
duplicate
work
right,
so
it's
totally
possible
to,
for
example,
and
I
dive
into
this
in
a
second,
have
the
monitoring
system
that
you're
already
using
provide
this
data,
and
so
this
is.
This
is
not
a
component
of
this
interest
special
interest
group.
Yet,
although
pretty
much
everyone,
the
special
interest
group
has
probably
touched
this
at
this
point,
or
at
least
somehow
influenced
it.
A
This
is
this:
the
kuba
dad
the
Prometheus
adapter
for
the
metrics
API,
and
essentially
this
is
exactly
what
I
was
just
saying.
There's
already,
you
may
already
be
running
Prometheus
and
it
may
already
be
collecting
all
these
metrics
from
your
containers,
and
so
it
may
be
unnecessary
to
run
a
second
component.
A
So
these
are
really
specific
to
your
monitoring
system
and
that's
why
there's
nothing
out
of
the
box
that
we
could
have
defined
here,
but
what
we
did
to
fine
is
the
API
for
this
so
again,
very
very
similar
scheme.
Essentially
any
monitoring
system
out
there
can
implement
these
and
I
think
pretty
much.
Every
major
monitoring
system
out
there
does
provide
an
adapter
for
this.
Prometheus
does
stackdriver.
Does
a
sure
does
I
believe
data
doc
does
pretty
much
pretty
much.
A
Every
monitoring
product
out
there
that
is
somehow
involved
with
kubernetes,
has
an
adapter
for
this,
and
so
the
custom
metrics
API
is
always
related
to
some
kubernetes
object.
So
you
say,
for
example,
in
your
horizontal
pot
autoscaler,
you
reference
some
deployment,
for
example,
and
so
the
custom
metrics
are
always
referencing
some
object,
whereas
there's
also
the
external
metrics
API,
and
that
really
doesn't
have
to
be
it's
very
similar,
if
you
know
philosophical
sense
through
the
custom
metrics
API,
but
it
doesn't
have
to
be
related
at
all
to
any
to
any
any
object
and
they're
really.
A
B
B
You
know,
change
this
and
like
people
who
realize
on
this
matrix,
like
no
matter
whether
like
they're
the
developers
administrators,
like
cloud
providers
or
you
can
keep
this
matrix
for
every
event
that
this
matrix
is
broken
and
there
was
an
idea
to
change
the
state.
This
idea
comes
from
Han,
so
honey
sitting
there
and
honey
is
going
to
talk
in
more
details
about
this
model.
B
B
As
a
part
of
this
metric,
we
want
to
migrate
control,
plane
metrics
to
the
framework
that
is
proposed
for
versioning,
which
means
the
metrics
that
are
existing
now
will
will
be
changeable
in
the
future.
We
will
do
this
once
so,
which
means
like
you
know,
and
and
then
and
then
we'll
have
like
a
proper
versioning
mechanism.
B
Okay,
so
I,
don't
know
whether
it
makes
sense
to
talk
in
more
details
now,
because
this
will
be
covered
by
Han
tomorrow
so
but
like
I,
think
one
will
think
that
what
to
mention
is
that
I
will
have
automated
check
in
the
kubernetes
repository.
We
actually
have
one
that
is
validating
the
changes
on
the
metrics
and
will
detect
if
something
is
done
in
non
non,
proper
way
and.
B
The
other
efforts
that
are
in
flight
in
there
on
the
additive
stage,
so
we
are
working
on
the
like
long
term
roadmap
like
three
years
roadmap.
Let's
say
starting
from
now
yesterday,
during
contributor
summit.
We
have
a
discussion
on
this.
Here
is
a
link
to
a
document
that
captures
more
or
less
this
discussion,
and
once
we
have
an
agreement
of
the
Stroud
map,
it
will
be
like
very
clear
what
district
will
be
working
for
the
next
two
years
and
as
a
part
of
this
roadmap,
we
have
a
few
ideas
to
do.
B
One
is
like
we
own
a
few
API
so
far,
they're
in
beta-
and
you
know
it's-
it's
not
a
good
state
when
the
API
Z
are
in
beta
for
ever
want
to.
You
know,
graduate
them
to
stable
or
decided,
maybe
they're
like
wrong
and
then
change
them,
but
it
doesn't
make
sense
to
keep
API
sin
beta
forever
in
not
like
as
a
part
of
this
effort,
we
also
want
to
release
version
v1
of
metric
server.
B
Metric
server
is
something
that
the
component
that
Fredrik
was
talking
earlier
about,
and
this
component
is
the
default
implementation
for
matrix
API.
That
is,
exposing
researchers,
add
metrics
for
kubernetes
system
for
kubernetes
pots
and
end
nodes,
and
you
know
this
is
also
in
kind
of
half-baked
state
we
want
to.
We
want
to
make
this
table.
There
are
also
two
other
fairly
large
efforts
like
which
is
like
introducing
structured
logging
and
introducing
tracing
for
kubernetes
objects.
So
those
are
efforts
that
are
in
the
discussion
phase.
There
is
some
proposal
of
a
cap.
B
This
is
still
under
review,
yeah,
so
how
you
can
get
envelope
when
T
of
people
asking
how
you
can
be
involved,
so
it
depends
on
first,
your
preferences.
What
you
like
to
do,
do
you
like
to
you
know?
Maybe
more
planning
conceptual
work,
or
maybe
you
want
to
be
more
involved
in
the
design
of
things.
Maybe
you
just
want
to
sit
down
and
code.
The
other
thing
is
like,
of
course,
like
your
experience
like,
depending
on
your
experience,
depending
on
your
expertise
like
you,
can
be
more
efficient
in
some
of
those
aspects
than
the
others.
B
Yes,
so
we've
been
talking
about
like
plenty
of
efforts
and
like
we
have
efforts
in
that
like
in
in
every
potential
like
we
have
efforts
in
various
stages,
which
means,
like
you
know,
the
matrix.
Our
whole
effort
is
in
execution,
so
I
feel
like
coding
like
this
is
like
a
good
effort
to
get
involved
because,
like
you
know
the
the
design
is
there,
we
know
what
to
do.
We
just
need
to
you
know,
invest
more
time
into
the
coding.
Yes,
the,
like
a
structured
logging
efforts.
B
Just
racing
effort
is,
like
you
know,
in
the
discussion
phase
in
the
design
phase.
So
if
you
want
to
get
involve,
if
you
want
to
review
this
is
like
a
good
effort
for
you.
If
you,
for
example,
like
planning
more,
like
you
know,
forward-looking,
like
defining
strategy,
come
join
us
into
this
effort
of
defining
the
roadmap,
how
to
reach
us.
So
we
have
a
meeting
every
turn
every
other
Thursday.
B
This
is
like
at
5:30
p.m.
UTC,
which
is
what
which
is
8:30
PST
some
local
time
here
for
me
6:30.
So,
like
you
know,
we
are
span
across
multiple
zones,
multiple
time
zones.
Here,
the
meeting
notes
we
have
a
channel
of
slag.
We
have,
of
course,
Google
Groups
discussion
group
for
for
further
discussions.
We
have
appropriate
groups
on
the
github,
so
you
can
like
see
see
groups
yeah
as.
C
B
B
Do
you
mean
this
structure
IRA
asking
about
the
structure,
locking
effort,
yeah
yeah,
so
the
those
efforts
are
about
introducing
some
frameworks
to
kubernetes
cold
ways
and
then
like
having
kubernetes
system
components
using
those
frameworks,
so
that,
like
you,
know
we'll
have
this
in
kubernetes,
but
I,
don't
see
any
reasons
why
that
everything
is
in
the
open
society.
Fink
is
openness.
It's
under
in
good
license,
Apache
one
and
you
know
I,
don't
see
any
reasons.
Why
not
reuse
this
by
you
know
other
systems.
B
A
So,
like
the
metric
stability
framework
that
that
we've
built
is
also
part
of
like
component
base
of
kubernetes
and,
like
I,
know
other
projects
outside
of
communities
that
are
you
that
use
component
base,
some
specifically
for
the
metrics
framework,
because
they
think
the
the
aspects
that
it
brings
are
interesting.
So
I
think
that
totally
a
possibility.
If
that's
what
you
want
to
do,
is.
C
B
A
Or
is
there
any
yeah,
so
renault
clio
state?
I
hun
and
I
had
a
bunch
of
discussions
about
the
global
metrics
registry
as
we
started
the
metrics
stability
framework
effort,
because
essentially
we
knew
that
if
we
start
this
effort,
we're
gonna
have
to
touch
every
metric
anyways,
and
so
we
had
a
bunch
of
discussions
and
I
think
we
we
did
end
up
agreeing
that
we
would
like
to
get
rid
of
the
global
metrics
registry,
because
it
has
brings
really
weird
aspects
to
it.
A
Where
suddenly
for,
for
some
reason,
you
have
api
server,
metrics
in
the
controller
manager
or
stuff
like
that,
which
is
really
odd
or
maybe
even
worse.
You
have
etsy
D
metrics
in
the
controller
manager,
and
so
that
calls
us
really
confusing
yeah,
because
it
causes
a
lot
of
confusion.
So
I
personally,
very
much
agreed
that
I
would
like
to
get
rid
of
this
global
metrics
registry.
We're
not
at
a
point
where
we
can
actually
do
that.
A
But
I
think,
as
we
continue
to
iterate
on
the
metric
stability
framework,
we
we
will
hopefully
one
day,
get
rid
of
the
global
metrics
registries
so
that
we
can
get
rid
of
these
weird
subtleties
that
that
the
global
state
brings
with
it
for
now
to
only
an
introduce
the
stability
aspect
of
the
metrics
we
have.
We
didn't
do
that
just
yet,
but
that's
totally
something
that
I'm
interested
in
doing
because,
like
I'm
feeling
the
pain
in
my
downstream
dependencies
as
well.
D
A
I
highly
recommend
going
to
the
deep
dive
tomorrow
because
there
will
be
basically
half
of
the
half
of
the
talk
will
be
about
try
saying
it's
still
very
much
an
early
state,
but
there's
some
really
interesting
work
being
done
in
this
space.
So,
yes,
we
totally
want
tracing
it's
difficult
to
inter
introduce
into
kubernetes,
because
it's
not
a
typical
RPC
style
system
right.
A
We
have
like
reconciliation,
loops
and
stuff
like
that,
and
that's
not
the
typical,
like
request
response
flow
that
like
tracing
was
nominally
made
for,
but
we're
told
we're
absolutely
interested
in
this
and
there's
work
being
done
and
I
think
yeah
just
definitely
go
to
this
talk,
because
obviously
it's
an
entire
talk
topic,
so
anything
that
I
would
say
is
not
enough
detail
and
probably
too
vague,
anyways
and
I.
Don't
do
this
work
so
I'm,
not
the
right
person
to
answer
so.
B
To
add
my
two
cents
to
your
question
like
if
you
are
thinking
about
those
two
free
things
like
you
know,
matrix
framework
like
locking
framework
tracing
framework.
Like
you
know,
the
earth
is
actually
this
starting
from
the
easiest.
I
was
like
you
know
this.
This
even
like
matrix
framework
was
like
very
hard
yes
and
like
the
cap
for
logging
for
structure.
Logging
has
like
20
pages,
so
it's
it's
not
easy
as
well.
Yes,
so
tracing
is
the
most
complex
here
and
you
know
we
started
from
what
was
easier
like
to
gain
experience
easier,
not
easy.