►
From YouTube: Kubernetes SIG Instrumentation: Instrumenting for Day Two Concerns - Marek Siarkowicz, Google
Description
Don’t miss out! Join us at our upcoming events: EnvoyCon Virtual on October 15 and KubeCon + CloudNativeCon North America 2020 Virtual from November 17-20. Learn more at https://kubecon.io. The conferences feature presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Kubernetes SIG Instrumentation: Instrumenting for Day Two Concerns - Marek Siarkowicz, Google
Kubernetes SIG Instrumentation is responsible for ensuring high quality and consistent instrumentation across the Kubernetes project. This intro session will give an overview of the efforts the SIG Instrumentation has worked on in the past and is currently working on. Software engineering and operations are both disciplines practiced in SIG Instrumentation, and any experience will help the special interest group's mission. Join this session to learn how to get involved in SIG Instrumentation to make Kubernetes' instrumentation even better!
https://sched.co/Zeub
A
A
So
we'll
start
from
the
sixth
chapter
and
we'll
talk
a
little
bit
about
the
kind
that
the
project
will
maintain
and
discuss
currently
ongoing
efforts.
At
the
end,
I
will
provide
you
with
some
thoughts
when
you
can
reach
out
so
on
the
next
slide.
You
can
see
that
we
will
start
from
the
charter,
so
kubernetes
is
a
big
project.
We
have
many
people
interested
in
different
areas,
areas
a
couple
specific
area,
community
forms
so
called
special
interests.
A
On
the
next
slide,
we
can
we
defined
what
is
in
our
scope
and
to
help
other
sikhs
in
their
instrumenting
their
features.
We
provide
advice
during
the
whole
process
of
designing
implementing
and
code
reviews
to
make
sure
that
community
complies
to
the
newest
guidelines
or
instrumentation
as
a
very
horizontal
seek,
and
we
touch
basically
every
component
in
kubernetes.
We
also
need
to
gather
requirements
from
other
sigs
allowing
us
to
design
common
apis,
so
they
can
be
used
by
everyone
and
so
on
the
next
slide.
A
We
we
have
something
that
are
outside
of
our
scope
and
we
don't.
We
don't
process
signals
outside
after
they're
exported
by
our
components
or
build
any
integrations
with
cloud
vendor
specific
solutions,
and
so
on
the
next
slide
we
can
see
our
team.
We
are
currently
19
members
and
we
have
selected
two
chairs
and
two
tech
leads
top
on
top
two
top
two
pictures
belong
to
our
chess.
It
terrorists
who
focus
on
organization
work
within
the
sikh
on
the
left.
We
have
han
frank
and
on
the
right,
elena
hashman,
on
the
bottom.
A
A
So,
on
the
next
slide,
I
will
have
a
list
of
all
projects
that
I
want
to
talk
about.
It
will
go
through
projects
from
the
earliest
one
to
the
newest
in
how
they
were
created,
so
some
projects
were
created
by
us
directly
and
and
other
like
cubesat
metrics.
We
adopted
to
make
sure
that
the
community
focuses
is
supported
by
the
community.
A
A
A
They
are
different
in
such
that
they
really
depend
on
the
type
of
application
that
we
that
is
running
so,
for
example,
requests
per
second
for
web
applications
or
q.
Lengths
by
workers
split
between
custom
and
external
metrics
exist
because
of
ability
of
referencing
kubernetes
objects.
So,
for
example,
to
simplify
our
configuration,
we
can
have
custom
matrix
reference
which
part
they
come
from
directly
on
the
next
slide.
We
can
see
core
matrix
pipeline
and
to
provide
metrics
out
of
the
box
in
kubernetes.
We
designed
this
pipeline
and
introduced
an
agent
called
matrix
server.
A
This
agent
will
scrape
metrics
core
metrics
from
nodes
and
sort
them
in
memory,
so
they're
available
to
be
served
in
api
and
on
the
next
slide
we
can
go
deeper
into
the
the
metric
server,
so
it
was
created
as
a
lightweight
solution
that
doesn't
require
any
persistent
storage
or
be
dependent
on
third-party
solution,
and
many
kubernetes
distributions
already
deploy
matrix
server
by
default
or
provide
an
easy
way
to
deploy
it
as
an
add-on,
and
so
on
the
next
slide.
We
can
see
the
pipeline
for
the
two
other
types
of
metrics.
A
Those
metrics
require
third-party
solution
to
provide
an
adapter.
This
adapter
would
translate
kubernetes
api
calls
to
a
third-party
native
query
language
allowing
to
expose
metrics
within
kubernetes
for
to
be
consumed
within
kubernetes
and
so
on.
The
next
site.
We
can
see
that
also
magic
server
can
be
totally
replaced
and
core
matrix
api
will
be
sourced
from
the
adapter
itself.
A
So
building
such
adapter
is
pretty
complicated
stuff.
So,
on
the
next
slide,
we
can
see
that
a
community
created
a
custom
matrix
adapter
and
it's
a
it,
helps
third-party
vendors
to
integrate
with
kubernetes
by
by
by
using
a
ready
library
that
contains
all
the
necessary
boilerplate,
and
this
simplifies
creating
an
adapter
to
just
implementing
a
couple
of
functions
in
go
language.
A
So,
on
the
next
slide
we
can
see
the
other
sick
project
which
is
cubistic
metrics
and
it
it
helped
checking
what
is
happening
in
the
kubernetes
cluster,
much
much
easier.
So
as
an
administrator
instead
of
running
kubectl
get
pots
every
few
seconds
just
to
check.
If
something
changes
you
can
just
deploy
keepstate
metrics
and
get
those
statuses
as
a
metric
and
ingest
them
in
your
monitoring
solution
and
its
main
purpose
is
to
read
all
the
kubernetes
state
of
kubernetes
objects
and
transform
them
into
premium
format.
A
A
This
makes
adding
pneumatics
very
easy
and
draw
a
lot
of
instrumentation
improvements,
but
also
led
to
a
lot
of
mistakes
and
over
the
years
the
lack
of
guidance
has
led
to
explosion
of
problems
with
with
monitoring
from
inconsistent,
naming
not
very
useful
selection
of
latency
buckets
up
to
security
issues
like
caused
because
of
memory
leaks
via
metrics.
That
could
be,
firstly,
create
created
from
exxonmar
externally.
A
So
on
the
next
slide,
we
can
see
the
our
effort
for
it
matrix
overhaul.
We,
we
started
this
effort
to
review
all
the
metrics
in
kubernetes
and
propose
fixes
to
them
to
bring
them
in
line
with
premises.
Guidelines.
Most
of
the
work
currently
is
finished,
but
we
are
still
finishing
the
final
paperwork
to
announce
the
ga.
A
A
A
A
We
could
not
have
maintained
them
all
so
also,
most
metrics
are
not
owned
by
sig
instrumentation
itself,
but
by
otherwise
we
could
not
empower
impose
the
cost
of
maintenance
costs
on
others
and
but
as
metrics
vary
in
maturity
importance.
We
we
needed
to
inform
users
and
ourselves
and
what
stability
we
and
them
can
expect
from
the
metrics.
A
But
after
the
matrix
vertical
metric
matures
enough
and
becomes
enough
that
it's
used
by
many
users
owning
sig
owning
this
metric
can
decide
to
make
a
commitment
to
maintain
it
and
graduate
it
to
stable,
stable
metrics.
We
define
for
stable
metrics,
we
defined
the
deprecation
policy,
and
this
the
policy
is
very
similar
to
what
we
do
for
kubernetes
cpi.
A
It
requires
any
change
to
to
to
go
any
breaking
change,
to
go
to
go
through
multiple
versions
until
to
maintain
previous
one
through
couple
of
versions
and
by
the
deciding
deprecator
metric.
We
automatically
will
notify
users
about
this
and
and
users
will
know
that
they
should
prepare
for
a
change
within
couple
of
versions.
A
After
the
application,
the
application
in
the
next
kubernetes
version
metric
will
be
hidden
by
default,
requiring
user
to
manually
re-enable
it,
and
only
after
two
versions.
A
metric
can
can
be
a
metric
that
was
marked
as
the
pre-cases
can
be
deleted
and
for
metrics
that
we
created
before
this
effort.
We
mark
them
as
an
alpha
metric,
but
still
we
we
want
to
go
through
a
duplication
period
to
allow
users
to
to
adjust
it
to
a
new
process.
A
A
On
this
slide,
we
can
see
how
we've
wrapped
prometheus
libraries
and
introduced
additional
fields
with
metadata,
such
as
stability
level
and
deprecation
for
users,
or
now
users
can
find
those
this
information
on
and
endpoints
expo
that
expose
those
metrics
or
by
checking
the
metadata
under
money.
Tolerance
solution
on
the
next
slide,
we'll
discuss
logging,
so
historically,
kubernetes
didn't
login,
didn't
change
a
lot.
Kubernetes
users
kubernetes
uses
own
library,
it's
called
k-lok,
it's
a
fork
of
old
google
library
which
is
not
under
active
development,
and
it
didn't
see
any
improvements
to
it.
A
Making
it
making
its
function
really
to
be
really
improved
when
we
compare
it
to
other
libraries
like
zap,
which
is
currently
pretty
popular
in
golang
community,
like
kylok,
has
really
similar
logging
format
of
printf,
which
makes
every
logline
totally
unique
and
very
cost
costly
to
parse
and
query,
and
so
we
introduced
a
new
effort.
So
on
the
next
slide,
we
introduced
an
effort
of
structured
logging
to
address
problems
mentioned
before
we
wanted
to.
A
Migrate,
kubernetes
locks
into
new
structured
form
and
we
started
it
by
defining
new
logging
functions
and
but
introducing
those
functions
was
not
enough
to
make
them
useful.
We
we
need
to
migrate,
also
migrate
old
old
locks,
so
they're
useful
to
to
users
but
kubernetes.
It
has
currently
around
more
than
10
000
log
calls
and
migrating.
Those
is
not
not
an
easy,
easy
task
and
to
make
a
new
login
format
useful
in
some
reasonable
time.
A
We
focused
on
maximizing
log
volume,
and
this
allows
us
to
keep
the
number
of
log
lines
to
be
migrated
small,
but
still
have
a
big
impact
on
how
logs
are
consumed
to
make
a
consumption
of
locks
much
easier.
We
also
introduced
a
new
json,
lock
format,
so
on
the
next
slide
we
can
see
main
changes
in
the
top.
A
A
We
could
not
change
the
header,
so
we
focused
on
changing
the
lock
message
and
each
argument
now
is
named
in
a
consistent
way,
allowing
for
creating
easy
tools
to
parse
it
and
on
the
bottom
right.
There
is
a
new
json,
lock
format,
which
is
much
much
easier
to
to
to
parse
and
doesn't
require
the
backward
compatibility
structure.
Logging
just
went
to
alpha
in
1.19.
A
So
now
we
can
go
to
another
purple,
another
proposal
in
login
on
the
next
slide.
We
will
discuss
lock
sanitization
so
last
year,
cncs
requested
a
security
audit
of
kubernetes,
and
this
audit
found
that
there
is
sensitive
data
like
tokens
and
secrets
locks
generated
by
kubernetes
and
because
of
size
and
complexity
of
the
project.
It
was
impossible
to
catch
those
things
during
code
review,
so
auditors
suggested
we
should
take
measures
that
would
prevent
to
store
this
sensitive
data
in
logs.
A
A
And
but
it's
very
very
useful
and
its
unique
feature
is
the
ability
to
to
trace
how,
for
example,
single
user
request
or
any
external
request
interacts
with
any
part
of
the
part
of
the
with
every
part
of
the
system
distributed
in
a
distributed
system,
and
we
can
with
tracing
we
can
find
which
microservices
where
participate,
participate,
participating
when
requests
were
was
processed.
How
long
the
request
spent
in
each
of
those
microservices
and
what
locks
were
generated
or
metrics
were
changed
or
impacted
by
it.
A
All
the
interactions
that
controllers
take
on
this
deployment
to
create
spots
and
then
those
spots
having
interaction
with
cubelet
and
being
started
by
other
containers
on
nodes,
and
this
looks
really
really
interesting.
I
would
encourage
you
to
look
into
that,
but
because
of
complication
on
api
machinery
site
and
the
first
proposal
was
changed
to
focus
on
the
standard
tracing,
but
it
opens
a
really
interesting
feature
for
bernatas.
A
A
A
A
A
It
can
read
metrics
from
kubernetes
cubelet
and
expose
them
to
to
to
to
auto
scaling
cube
state
metrics
works
differently.
In
that
it
reads:
it
exposes
metrics
in
premises,
format
allowing
your
monitoring
solution
to
use
them
externally,
and
so
to
summarize,
metric
server
is
more
for
auto
scaling
and
chip.
State
matrix
is
more
for
application,
alerting,
observability
of
deployment
and.