►
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Kubernetes SIG Instrumentation - Deep Dive - Han Kang & David Ashpole, Google
Kubernetes SIG Instrumentation is responsible for ensuring high quality and consistent instrumentation across the Kubernetes project. This deep dive session will go in detail currently ongoing efforts happening within SIG Instrumentation to share with the audience concrete pieces of work to encourage future collaboration. Specifically, we go into our ongoing efforts with the Kubernetes metrics stability framework and discuss our current exploration into adding distributed tracing to Kubernetes objects and their lifecycles. Software engineering and operations are both disciplines practiced in SIG Instrumentation, and any experience will help the special interest group's mission. Join this session to learn how to get involved in SIG Instrumentation to make instrumentation even better!
https://sched.co/Uagv
B
A
B
B
A
B
A
A
A
A
B
A
B
B
Features
so
this
is
clearly
better
things
to
do
other
ideas
that
we've
thought
of
the
past,
but
and
the
pursuit
was
what
was
a
confrontation
image
that
has
all
the
metrics
of
kubernetes
and
explicitly
list
what
the
guarantees
but
I,
don't
think
we
would
be
able
to
do
that
very
well
keeping
so
either
of
those
really
work.
He
doesn't
mean.
A
A
A
Always
has
these
three
steps,
as
you
can
see?
First,
you
define
a
metric
in
this
case.
We
have
a
summary.
We
have
a
name,
we
have
help
and
we
have
quantile
information.
Second,
we
take
that
metric
definition
pass
it
in
to
a
constructor
Prometheus
knew
some
rebec
would
be
the
function
call
and
we
instantiate
that
metric
and
we
we
retain
a
reference
to
that
metric
instance
and
then
somewhere.
A
We
register
this
thing
or
enroll
this
to
a
metric
registry,
and
that
allows
this
metric
to
be
published
to
a
metrics
endpoint
and
we
take
the
reference
from
that
singleton
somewhere
in
our
instrumented
code,
as
you
can
see
on
the
bottom,
and
we
actually
observe
some
sort
of
thing
that
we
want
to
measure.
So
the
fact
that
this
that
the
metric
life
cycle
always
occurs
with
these
three
steps
gives
us
an
opportunity
and
it
basically
gives
us
an
opportunity
to
inject
our
own
custom
logic
around
the
life
cycle.
A
A
Obviously
these
things
will
not
do
anything
in
it
of
itself
right.
You
can't
just
add
a
couple
of
fields
and
expect
things
to
work.
So
next
you
have
to
hijack
the
instantiation
right,
so
we
wrap
all
of
the
Constructors
which
taken
now
these
new
extra,
these
new
definitions
and
instantiate
a
metric
with
that
metadata.
A
Lastly,
we
hijack
the
mystery
Registry,
so
we
have
a
wrapper
struck
which
embeds
of
Prometheus
registry
and
we
can
inject
custom
logic
during
the
registration
cycle.
So
this
allows
us
to
do
a
lot
of
interesting
things.
Work
a
lot
of
registration
magic
depending
on
the
stability
level
of
of
a
metric.
A
So
now
that
you've
seen
the
stability
levels
and
the
deprecation
the
the
deprecation
field,
you
can
see
that
we
have
two
stability
axes
accesses
and
the
stability
classes
are
either
alpha
or
stable
and
for
alpha
metrics.
Basically,
what
we
are
saying
is
that
there
are
no
stability
guarantees,
it's
not
a
stable
API
for
stable
metrics.
These
things
are
guaranteed
not
to
change.
So
now
you
can
instrument
your
dashboards
or
alert
based
off
of
this
thing.
A
Next
we
have
the
deprecation
axis
where,
basically,
what
we
can
do
is
we
can
signal
the
future
definition
of
a
metric,
and
so
it
looks
like
this.
If
you
have
a
stable
metric
in
115,
let's
say
you
deprecated
in
116,
so
you
toggle
this
bit.
Deprecated
version
to
1,
116
you're,
basically
saying
this
thing
is
going
to
be
removed
in
the
subsequent
release.
That
metric
will
be
automatically
hidden
from
the
endpoint.
So
let's
say
you
are
ingesting
metrics
at
this
point
and
at
this
point
you've
upgraded
your
cluster.
A
A
Second,
we
added
static
analysis
to
validate
that
each
of
these
things,
Don
Val,
did
not
break
our
stability
guarantees
in
the
beta
stage
of
the
metric
stability
framework
were
gonna
forbid,
the
direct
use
of
Prometheus
in
kubernetes
components,
which
basically
means
everything
has
to
go
through
the
metric
stability
framework.
So
this
prevents
people
from
making
errors
for
the
GA
release
of
the
metric
civility
framework,
we're
basically
providing
an
escape
hatch
so
that,
let's
say
we
have
a
stable
metric,
which.
A
A
And
lastly,
the
nice
thing
about
the
stability
framework
is
that
this
has
allowed
us
to
consolidate
a
bunch
of
this
code
into
a
single
repository,
that's
actually
owned
by
saved
instrumentation,
which
has
had
side-effect
of
allowing
us
to
sort
of
consolidate
and
centralize.
What
is
otherwise
been
a
horizontal
effort.
B
So
now,
I
get
to
tell
you
about
the
fun
I've
been
having
trying
to
use
tracing
in
kubernetes
controllers,
and
this
has
been
something
that
I've
been
working
on
for
about
a
year
now,
just
kind
of
in
my
free
time,
but
I
think
it's
really
fun
and
a
demos
really
well
to
be
honest.
So
just
to
give
a
super
quick
overview
on
tracing
the
standard.
B
B
To
give
you
a
nice,
pretty
picture
like
something
like
this
and
then
the
cool
thing
about
this
is
I
can
take
one
look
at
it
and
know
immediately
that
the
rights
to
backend,
one
and
back-end
two
are
sequential,
not
in
parallel
because
of
the
structure
of
the
graph.
So
it's
a
nice
way
to
not
only
collect
information
and
tie
things
together
from
different
components,
but
also
to
be
able
to
visualize
what's
happening
in
our
system,
and
we
call
this
little
box
here
a
span
and
the
whole
thing
together
at
trace.
B
So
to
start
with,
I'm
gonna
tell
you
guys
a
quick
story
of
a
bug
that
I
had
about
this
time.
Last
year,
pod
startup
normally
takes
about
three
seconds.
We
had
a
case
where
it
would
take
more
than
50
seconds
sometimes,
and
that's
pretty
bad.
So,
like
all
good
engineers
who
had
a
hard
problem,
I
got
an
intern
and
gave
the
problem
to
him,
and
so
he
worked
on
this
for
three
or
four
months
and
he
he
had
some
fun
with
it
and
got
to
apply
tracing
to
this
problem.
B
To
help
give
us
more
insight,
and
so
these
are
some
of
the
initial
traces
that
I
dug
up
of
stuff
that
he
did.
You
can
see
here-
or
you
can't
see
here,
but
the
big
blue
box
says
38,000
milliseconds,
which
is
about
38
seconds,
and
we
were
able
to
track
this
bug
down
and
found
out
that
it
actually
was
an
entropy
bug
in
our
operating
system.
B
So
when
the
node
first
boots
up
doesn't
have
enough
entropy
to
generate
a
container
ID,
and
so
it
hangs
until
it
can
it's
fixed
now,
thankfully,
but
that
sort
of
planted
the
seed.
In
my
mind
and
I
said
wait.
This
would
be
really
really
cool
to
be
able
to
use
to
trace
other
things
in
kubernetes,
because
right
now
things
aren't
so
great.
So
let's
take
a
look
at
what
we
have
in
kubernetes
right
now
and
sort
of
where
we're
lacking.
B
Usually
my
go-to
ends
up
when
I
debug,
the
production
issues
in
gke
ends
up
being
blogs.
So
you
go
in.
You
go
to
the
cubit
logs,
you
grep
for
the
name
or
grab
for
the
UID
or
grab
for
the
container
ID
right
and
it's
a
big
struggle,
because
the
cubit
logs
a
lot,
and
so
this
is
all
slow
and
it's
no
fun
so
logs
are
there
but
they're
kind
of
a
last
resort.
B
Metrics
are
awesome
when
you
have
a
cardinality
that
gives
you
the
metadata
that
you
need
so
most
of
the
time,
it's
not
feasible
to
put
things
like
a
container
ID
in
a
metric,
because
that
would
explode
your
back-end
and
events
are
awesome
but
they're
short-lived.
So
for
me,
when,
at
the
point,
when
I
usually
get
to
a
customer's
cluster,
they're
all
gone
and
once
I
have
them,
they
can
kind
of
be
difficult
to
parse.
B
So
I
I
can
see,
for
example,
that
there
are
a
bunch
of
events,
all
of
which
we
have
ten
thousand
of,
and
so
I
know
something's
going
wrong,
but
unless
I
really
know
what
I'm
doing,
which
most
of
the
time
I
do,
I
can't
figure
out
what's
actually
wrong
in
the
system
and
track
it
down.
So
my
favorite
part
about
tracing,
though,
is
just
how
easy
it
is
to
visualize
and
look
at
you'll,
see
later
in
my
demos,
that
tracing
is
really
a
nice
way
to
present
information
and
as
a
kubernetes
developer.
B
I
really
want
this
for
myself.
Okay,
so
I've
been
talking
about
kubernetes
as
if
the
standard
RPC
model
doesn't
apply,
but
technically
there
is
one
place
where
it
does
and
that's
the
API
server,
so
user
sends
requests
to
the
API
server
to
create
a
config
map
that
then
gets
forwarded
to
sed.
It
comes
back
and
we're
all
done,
and
so
we
would
expect
that
request
to
look
something
like
this
here,
so
I
figured
I
would
give
this
a
quick
demo.
B
B
B
And
all
I'm
doing
here
is
attaching
a
context
to
this
request
with
my
dash
dash
trace.
So
then,
if
I
pop
over
to
Zipkin
I
can
go
look
for
my
traces
and
lo
and
behold
there's
a
trace.
That
looks
exactly
like
what
I
expected
it
to
I
sent
my
request
to
the
API
server,
got
it
from
a
CD
and
that's
pretty
cool
right.
If
you're
Han
over
here
who
works
on
the
API
server
you're
happy.
If
you
everyone
else,
then
there's
still
some
stuff
missing.
So,
let's,
let's
go
dive
back
into
that
great
great
ok.
B
B
B
So
for
HTTP,
for
example,
in
order
to
send
a
context,
that's
in
a
context
context
go
object
from
one
go
binary
to
another.
What
you
have
to
do
is
you
have
to
take
the
value
out
of
the
context.
Stick
it
in
the
request.
Header
send
it
along
with
the
request
and
then
the
receiving
server
can
take
it
out
of
the
header
and
then
put
it
back
in
a
go
to
go
context
for
me
to
use
with
my
geometry
so
for
kubernetes.
B
We
want
to
do
something
that
looks
pretty
similar
to
this
and
actually
there's
something
that
fits
the
bill
pretty
well,
instead
of
using
HTTP
header,
which
are
key
value
stores,
we
can
use
this
lovely
key
value
store
that
kubernetes
gives
us,
so
we
can
take
the
context
out
of
the
context.
Stick
it
in
an
annotation
and
then
the
next
controller
that
works
on
that
object
can
take
it
out
of
the
annotation
and
use
that
to
instrument
their
traces
all
right.
So
as
a
node
developer,
I
said:
okay,
let's
start
with
a
pod
right.
B
B
B
B
Okay
cool,
so
this
looks
like
our
trace.
So
let's
take
a
look
at
this.
I
can
see
that
just
like
when
I
created
my
config
map.
I
have
this
these
initial
spans
here
that
are
for
creating
the
object
for
the
pod
and
the
sed
transaction
for
that.
But
then
I
can
see
that
I've
got
a
whole
bunch
of
other
stuff
too.
I
can
see
that
the
cubelet
or
sorry
the
scheduler
is
the
next
guy
up,
and
we
can
see
that
the
pod
is
scheduled.
B
We
can
see
that
the
cubit
updates
the
status
to
reflect
that
it's
received
the
pod,
and
then
we
see
all
of
the
fun
container
runtime
operations
that
are
being
sent
to
docker.
So
all
of
this
we
can
now
see
connected
together
across
multiple
different
kubernetes
component
for
the
pot
object,
and
that
was
that
was
pretty
cool.
Being
able
to
get
to
this
point
as
honset.
B
B
Just
like
in
the
API
server
I
had
to
add
something
to
wrap
the
or
I
had
to
add
a
wrapper
around
the
HTTP
client
I
had
to
add
a
G
RPC
dial
option
for
the
container
runtime
interface
in
the
scheduler
I,
tossed
in
a
start
span
from
object,
which
is
just
a
helper
function.
I
define
to
take
the
context
out
of
the
object
and
start
a
span
with
it,
and
this
is
all
I
had
to
do,
was
wrap
what
I
was
doing
in
a
span
and
then
in
the
cubelet.
B
I
had
to
do
almost
exactly
the
same
thing
when
it
decides
that
it's
going
to
create
the
pod
I
had
to
start
a
span
from
the
object,
but
then
that
context
that
I
get
back
from
starting
the
span.
I
can
actually
use
in
all
of
my
calls
to
the
container
run
time
to
get
instrumentation
from
those.
So
all
of
that
was
added
just
just
required,
adding
a
couple
more
lines
of
code
to
get
all
all
that
fun
stuff,
okay
and
then
back
to
the
thing.
I
was
initially
trying
to
do
right.
B
I
want
to
create
a
deployment,
but
this
is
slightly
more
complex
because
it
spans
multiple
objects.
It's
not
just
here
create
this
object
attach
of
context
now,
there's
sort
of
this
weird
hierarchical
relationship
between
some
and
I
need
to
figure
out
how
to
deal
with
that.
So
I
could
just
say:
hey
I'm,
a
node
guy,
all
I
really
care
about
the
cubelet.
Just
give
me
a
bunch
of
podraces
because
after
all,
that's
all
deployment
really
is.
B
But
my
friends
who
work
on
the
controller
manager
would
disagree
with
me,
so
I
think
it's
more
useful
to
have
one
trace
per
object
that
you
create.
That
describes
all
the
things
that
happen
in
kubernetes.
So
what
we
really
need
to
be
able
to
do
is
to
figure
out
a
way
to
propagate
context
from
the
higher-level
objects
like
a
deployment,
a
replica
set
down
to
things
like
pods.
B
B
All
right,
so
we
created
the
deployment
nots
go
back
to
zip
kin,
see
what's
in
here,
okay,
so
this
latest
one
I
think
is
what
we're
looking
for
I
like
to
collapse
them
all.
First,
okay!
So
let's
walk
through
this
tree
really
briefly,
just
to
see
all
the
goodness
that's
inside.
So
we
have
the
initial
right
just
like
before
a
TD
transaction
awesome.
Here
we
can
see
that
the
deployment
has
added
a
specific
trace
for
creating
the
replica
set,
which
is
not
happy
with
me.
B
Each
of
these
right
we
can
see
all
of
the
work
from
all
of
the
nodes
in
my
cluster
is
required
to
create
each
of
the
pods,
and
this
you
can
start
to
tell
where
this
abstraction
starts
to
get
really
powerful
and
really
really
useful
and
I'm
super
hopeful
that
this
can
be
something
that
is
useful
to
others,
as
well
being
able
to
see
all
the
way
from
the
Etsy
D
transaction.
That
creates
an
object
all
the
way
down
to
the
the
calls
to
the
CRI
that
are
doing,
pull,
image
and
and
all
that
fun
stuff.
B
B
Continuing
on
what
did
I
have
to
do
to
to
make
that
work.
Well,
it's
actually
not
that
hard
I
had
to
do
pretty
much
the
same
thing
as
I
did
last
time,
except
you
can
see
I
so,
for
example,
for
the
deployment
controller,
I
added
a
span
for
creating
the
replica
set
and
then,
but
instead
of
taking
that
context
and
passing
it
to
say
a
G
RPC
call
to
the
container
on
time,
interface,
I'm.
B
Taking
that
context
and
sticking
it
in
this
object
that
I
need
to
create
right
so
right
before
I
create
my
replica
set
I
just
say:
replica
set.
Here's
your
context,
and
now
all
the
things
associated
with
that
replica
set
are
tied
back
to
my
deployment
creation,
which
is
really
cool,
same
thing
for
the
replica
set,
creating
a
pod
right,
same
steps
start
a
span
and
then
just
stick
the
context
inside
of
our
object.
B
Okay,
now
all
good
engineers
can
take
a
deep
breath,
we're
going
to
generalize
this
a
little
bit
right
and
try
and
figure
out
what
the
patterns
are
and
how
we
can
apply
this
to
more
than
just
David's
deployment.
That
is
really
simple.
So
what
things
should
we
wrap
in
a
span,
and
this
gets
a
little
bit
tricky
in
kubernetes,
because
there's
lots
of
like
background
processes
that
are
constantly
reconciling
and
we
probably
don't
want
to
trace
all
of
that.
B
It's
the
important
thing
that
I've
found
is
that
we
only
want
to
export
spans
when
we're
doing
real
work.
So,
for
example,
if
I
had
a
cube
builder
controller
here,
that
is
reconciling
on
a
my
object.
I
want
to
make
sure
that
I
have
this.
If
updates
required
before
I
start
my
spans,
otherwise
I'm
just
gonna
get
a
span
each
time
I
sync
or
see
irrelevant
updates.
So
if
I
remove
this,
that
would
be
bad
and
I
would
get
a
bunch
of
random
spans
that
don't
make
any
sense.
B
I
forgot
something
and
then
which
object
should
I
should
that
span
be
associated
with
well
actually,
using
cube
builder
makes
this
kind
of
obvious,
because
usually
I
get
a
reconcile
and
that
reconcile
is
associated
with
a
specific
object
right.
So
there's
some
object
that
isn't
in
its
desired
state
that
I
need
to
move
to
its
desired
state,
so,
for
example,
with
replicas
set.
B
B
So
now
to
the
question
of
like
okay,
I've
got
my
hierarchical
objects
and
one
I
need
to
create
one
in
order
to
make
another.
How
does
that
relationship
work
and
when
should
I
propagate
spans
there?
Well,
it's
actually
almost
exactly
the
same
answer
as
the
question
before
when
we're
doing
work
that
moves
an
object
from
its
from
a
current
state
to
its
desired
State.
Those
are
the
types
of
things
that
we
want
to
trace
and
if
it
just
so
happens
that
that
work
involves
creating
another
object.
B
Well
then
that's
the
work
that
we're
tracing
so
again,
with
my
cue
builder
example,
when
I'm
doing
a
reconcile
on
some
object
and
in
order
to
bring
that
object
to
its
desired
state,
I
need
to
say,
create
another
or
update
another.
Well
then,
that's
when
I
should
propagate
my
context
to
that
child
object
and
I
want
to
highlight
to
that.
This
is
a
total
work
in
progress.
None
of
this
is
merged
up
scream,
there's
a
kept
for
it.
That's
been
open
for
a
long
long
time
has
had
lots
of
lovely
discussion
on
it,
but.
B
There's
definitely
room
for
for
lots
of
other
people
to
contribute
and
to
bring
their
own
ideas
and
to
help
move
this
forward
so
that
we
can
all
get
the
that
cool
experience
that
you
got
earlier
when
we
were
like
wow.
We
can
see
everything
that's
happening
to
a
deployment,
but
I
just
want
to
mention
a
couple
of
the
fun
hard
problems
that
still
don't
have
complete
solutions
to
them.
One
is
normally
an
RPC
systems
trace
ends
when
everything
returns
right.
B
B
I
can
make
rights
to
a
database
right,
it
all
just
works,
but
in
kubernetes
if
he
updates
the
replicas
count
in
our
deployment
and
then
I
come
in
afterwards
and
update
the
image.
Well,
the
controllers
are
just
reconciling
to
some
desired
state,
and
should
they
associate
things
that
they're
doing
with
what
he
said
to
do
or
with
what
I
said
to
do
right.
So,
at
the
end
of
the
day,
we
can
do
a
couple
different
things.
B
Maybe
we
could
try
and
do
both
like
he
gets
a
trace
and
I
get
a
trace,
but
that
seems
like
it
might
have
some
scaling
issues
or
other
problems.
The
other
thing
we
can
do
is
say
that
well,
we're
gonna
tie
everything
to
ever
made
the
last
update,
but
then
he's
left
out
in
the
cold
if
he
would
like
to
know
what
what
happened.
So
maybe
we
can
link
the
traces
and
maybe
that's
enough,
but
there's
definitely
a
lot
more
interesting
stuff
to
be
solved
there.
B
So
what
does
this
look
like
so
I'm,
working
on
the
cap
and
I've
seen
some
excitement
around
it
recently.
So
I'm
very
optimistic
at
this
point,
but
if
and
when
that's
approved,
there's
a
ton
of
work
to
do.
Imagine
if
I
were
standing
up
here
telling
you
about
the
merits
of
using
metrics
and
kubernetes,
like
there's
so
much
code
as
Han
described
earlier,
that's
dedicated
to
monitoring
inside
of
kubernetes
and
there's
a
ton
of
work
that
needs
to
be
done
to
instrument
all
of
the
things
that
exist
right.
B
You
add
them
one
day
and
then
they're
out
of
date,
the
next
and
because
you're
just
a
component
owner
and
can't
don't
have
the
time
to
maintain
20
integrations.
They
all
sort
of
dissolve
and
it's
a
generally
a
bad
thing,
but
I've
been
super
excited
by
open
telemetry
and
the
projects
that
came
before
them,
because
they
sort
of
offer
this
this
promise
of
being
able
to
have
telemetry
inside
kubernetes
that
isn't
specific
to
any
particular
back-end
and
that
actually
works
well
with
all
of
them.
B
So
I
think
this
is
a
great
place
for
a
great
tool
for
us
to
use,
because
we
can
stay
vendor-neutral
inside
of
the
kubernetes
components
without
needing,
and
but
we
can
still
support
in
a
really
solid
way.
I
hope
all
of
these
vendors
so
to
end
I
have
one
last
demo,
which
is
that
all
of
the
stuff
I've
done
I've
also
been
sending
to
Jaeger.
So
we
can
see,
for
example,
the
deployment
the
same
deployment,
tracing
Jaeger
right
and
I've
also
been
sending
everything
to
stack
driver.
B
A
B
Okay,
so
caps
generally
one
so
I'll
talk
from
personal
experience,
because
I
worked
on
five
or
six
of
them,
the
bigger
your
kept.
It
gets
exponentially
harder
to
get
it
merged,
because
if
I
have
a
kept
and
I
need
to
get
it
approved
by
one
sig
well,
then
I
can
just
go
to
that
SIG's
meeting
every
week
and
bother
them
until
they
answer
all
my
questions
and-
and
we
can
agree
on
something
right.
B
But
what
tends
to
happen
in
really
big
caps
is
that
you
go
to
one
sig
and
they're
like
okay,
make
this
change
and
then
we're
good,
and
then
you
go
to
another
sig
and
they're
like
no,
no
revert
that
change
right.
So
it's
hard
to
get
the
SIG's
to
talk
to
each
other.
Sometimes
so
this
particular
one
mainly
involves
sig,
instrumentation
and
I
would
say
what
does
client
go
fall
under
API.
A
B
Machinery
and
I'm
also
making
changes
to
keep
control
so
probably
six
CLI
right.
So
there's
a
few
just
a
couple
SIG's
that
I
need
to
go,
get
on
board
and
I'm,
not
making
any
API
changes,
which
is
a
big
like.
If
you
want
to
get
things
in
quickly,
how
can
you
do
it
without
making
a
p-type
API
changes?
So
I
expect
this
one
shouldn't
take
too
long
compared
with
some
of
the
other
big
stuff
that's
been
going
through
recently.
B
A
Anyone's
interested,
we
can
probably
make
the
client
code
changes
orthogonal
to
this
thing,
so
it
can
be
completely
decoupled.
We
just
need
to
pass
context
through
all
the
Franco
methods,
and
you
could
do
this
in
like
a
backward
compatible
way.
But
if
someone
wanted
to
write
a
cap
on
that
and
like
work
on
that
thing,
then
that
would
make
landing
this
thing
like
it's.