►
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Deep Dive: Kubernetes (Instrumentation) SIG - Frederic Branczyk & Max Inden, Red Hat
The SIG Instrumentation deep dive is going to cover some of the details of performance optimizations we have recently been working on, both on a technical level of how they were achieved, but also the results of said optimizations. Beyond that we will dive into advanced topics of how to make use of the data Kubernetes exposes for advanced querying in order to perform monitoring and alerting on Kubernetes clusters.
https://sched.co/MPjo
A
Okay,
I
guess:
let's
get
started
thanks
everybody
for
coming
to
the
second
cementation
deep
dive.
In
case
you
missed
the
intro,
it's
recorded,
but
just
a
very
general
overview
of
sig
instrumentation,
we're
basically
responsible
or
India
in
charge
of
the
instrumentation
pieces
in
kubernetes.
That
doesn't
necessarily
mean
we
own
all
the
lines
of
code
that
do
metric
or
log
or
tracing
instrumentation,
but
it
means
that
we
provide
the
best
best
practices
and
guidelines
for
the
project
for
other
SIG's
to
hopefully
comply
to
and
frameworks
for
these
things
and
yeah.
A
If
you
want
to
know
more
of
the
basics
of
that
check
out
the
second
fermentation
intro,
we
are
at
the
deep
dive
here
and
actually
max
is
going
to
be
talking
about
one
of
our
sub
projects,
so
yeah
I'm
not
actually
part
of
this
session,
even
though
I'm
kind
of
in
the
schedule,
but
take
it
away
max.
Thank
you.
B
All
right
cool
so
after
this
introduction,
thank
you.
I
want
to
talk
about
optimizing
metric,
rendering
in
keep
statement
ryx,
which
is
one
of
the
components
of
sec,
instrumentation
and
now
we'll
go
a
little
bit
into
what
that
is,
and
what
that
means,
and
so
on
mostly
metrics
rendering,
from
the
perspective
about
like
the
Prometheus
metric
format
and
how
we
can
do
that
in
a
very
efficient
way.
B
So
how
can
we
performance
optimize,
metric,
rendering
so
in
the
end,
string
interpolation
in
a
very
efficient
way
for
myself
and
max
and
part
of
RedHat
and
part
of
the
Prometheus
upstream
team
as
well?
Some
one
of
the
maintainer
of
a
load
manager,
for
example,
but
also
I'm
very
connected
to
the
Camaro's
world,
so
whatever
connects
the
two
worlds
and
I'm
working
on
the
bunch,
yeah
cool
all
right,
so
we
already
saw
this
light.
B
Optimizing
metric
renewing
of
CubeSat
metrics
on
that
of
that
very
specific
component
in
second
cementation.
But
you
could
also
take
a
different
perspective
at
this
talk
or
look
at
it
differently.
Maybe
you're
not
very
interested
in
cube
state
metrics,
for
example,
or
that
particular
component
or
you're,
saying
okay,
it
performs
optimized
now
like
cool
I,
don't
really
need
to
know
about
the
internals,
but
the
alternative
perspective
you
can
really
have
here
is
how
can
I
performs
optimize
an
application
with
the
help
of
my
monitoring
system
on
top
of
kubernetes?
B
That
is
the
other
perspective,
but
still
I
look
at
it
at
this
perspective
of
cube
state
metrics.
So
whoever
heard
of
CubeSat
metrics
right,
we
are
very
used
to
keep
state
metrics.
That
is
cool,
okay,
all
right,
so
we
have
a
problem
that
we
want
to
solve.
With
keep
state
metrics.
We
have
the
API
server.
On
the
one
hand,
the
API
server
or
kubernetes
in
general,
has
a
bunch
of
state
that
is
mostly
saved
inside
of
EDD,
and
we
want
introspection
in
that
into
that
state.
B
We
want
observability
into
that
state
and
we
don't
only
want
snapshots
of
that
state,
but
we
also
want
that
in
history
like
in
a
time
serious
manner,
kubernetes
very
nice.
It
gives
us
great
API
on
top,
but
it
can
really
only
give
us
the
current
view,
and
it
also
only
gives
that
to
us
in
in
in
the
end,
in
a
restful
manner
as
kubernetes
objects.
On
the
other
hand,
we
want
to
funnel
all
this
into
prometheus
Prometheus,
there's
a
time
series
database
or
any
really
any
of
your
monitoring
systems.
B
This
expects
a
certain
format,
so
in
object-oriented
programming.
This
would
really
like
just
be
the
adapter
pattern
and
permit
us.
This
is
called
the
exporter
pattern,
so
we
need
something
in
between
the
two
to
translate
the
tree
and
that
would
be
cube.
State
metrics.
Here's
like
a
small
scenario.
You
have
the
API
server.
We
use
the
watch.
Api
notify
us
that
there's
new
pod,
we
take
that
part.
B
We
cache
it
inside
of
memory
on
cube,
state
metrics
and
then
a
synchronously
or
not
in
line
with
that
Prometheus
comes
along
at
some
point,
tries
to
get
all
the
metrics
it's
on.
Pull
model
so
it
does
request,
cube
state
metrics
iterates
over
its
entire
cache
takes
every
part,
looks
at
it,
generates
the
metrics
out
of
those
parts
or
contract
maps
or
whatnot,
and
then
returns
to
the
metrics
as
and
in
the
end,
it's
a
byte
slice
on
golang
but
yeah
anything
really
cool.
This
is
the
architecture
of
the
old
version.
B
The
overall
idea
stayed
the
same.
It's
still
the
adapter
between
the
two,
but
the
internals
changed
quite
a
bit,
so
I
refer
to
the
old
version
as
the
not
so
much
it
performs,
optimized
versions
and
with
the
recent
performance
optimizations.
That's
the
new
version
that
we're
going
to
talk
about
cool
a
little
bit
more
like
what
do
we
actually
cache
or
what
am
I
talking
about
when
I
talk
about
commit?
B
Nets
objects,
you
all
know
the
Yammer
representation
here,
for
example,
you
have
a
pod
all
that
info
you
would
get
from
the
rest
of
all
API
and
on
the
bottom
you
see
a
snippet
of
the
metrics
that
we
would
generate
out
of
this
nag.
For
example,
we
have
a
coupon
container
info
metric,
which
tells
you
the
container
name
namespace
and
the
pod,
and
you
really
really
see
the
data
matching
like
it's
really
like
traversing
one
data
into
some
other
in
general.
B
Cube
state
metrics
doesn't
do
any
heuristics
on
the
data,
but
tries
to
give
you
the
raw
data
so
that
you
can
do
all
the
calculations
and
allows
us
inside
your
monitoring
system.
That
makes
you
most
powerful
on
that
and
all
right.
So
we
cache
the
kubernetes
object
and
we've
returned
prometheus
metrics
to
Prometheus
in
the
end.
B
Now
all
of
this
worked
quite
well.
Actually,
I
can
go
love
it
back
one
more
time
all
the
communication
between
API
server
and
chip
state
matrix
was
entirely
happening
via
kubernetes
client
go
and
then
all
the
communication
between
cube
state,
matrix
and
Prometheus
was
entirely
happening
with
Prometheus
client
go,
so
we
promptly
just
had
to
change
the
prefix
and
Oliver
was
just
figured
out
for
us.
Now
we
had
some
problems
with
this
design.
It
worked
quite
well
over
time,
but
long
term
problems
were
on
bigger
clusters.
B
We're
talking
about
something
like
50
megabytes
of
the
metric
output
or
higher.
So
that's
I,
don't
know
our
benchmarking.
Clusters
were
at
least
250
nodes,
constantly
scaling
workloads
so
that
it
was
a
small
cluster,
but
we
had
customers
on
a
lot
bigger
clusters,
so
problems
we
had
were
high
response
rates,
we're
talking
about
10
to
20
seconds
about
simply
like
Prometheus
during
the
request
and
then
the
metrics
coming
back
that
whole
time
span
was
20
seconds.
B
In
addition,
like
this
in
terms
of
business,
in
terms
of
monitoring
data
like
having
all
your
monitoring
data
20
seconds
late
is
a
problem
because
you
might
want
to
correlate
that
data
with
other
systems
and
if
some
are
20
seconds
off
and
some
are
not
or
some
are
30
seconds
off,
correlating
the
two
is
very
difficult,
and
that
would
be
the
power
of
Prometheus
or
any
monitoring
system
to
correlate
different
data
streams,
which
we
really
want
to
do
with
keep
state
metrics.
So
we
want
to
reduce
that
number
by
a
lot.
B
In
addition
running
this
on
bigger
clusters,
we
had
quite
a
lot
of
issues
opened
about
like
how
much
memory
does
this
actually
use
how
much
CPU
should
I
give
it.
So
all
of
this
planning
is
quite
difficult
in
general,
we
could
optimize
it,
but
also
one
goal
was
to
stabilize,
for
example,
CPU
usage
by
a
lot,
so
not
have
it
be
a
very
spiky
workload,
but
instead
being
quite
flat
out
right.
So
that's
the
problem
statement
now
this
was
all
inside
of
the
instrumentation,
so
we
do
this
well.
B
Of
course
we
use
our
monitoring
system
to
optimize
our
monitoring
system.
So
let's
look
at
a
couple
of
metrics
that
we
want
to
use
to
optimize
all
of
this
or
the
metrics
that
we
optimize
one
optimize
for
so
the
first
thing
is
really
squared
duration
seconds.
This
is
just
a
snippet
least.
I
did
later
on
after
the
performance
tested
things,
I
didn't
have
the
bigger
cluster
available
anymore,
there's.
Actually
a
super
small
cluster
was
a
huge
amount
of
config
maps,
not
very
optimized,
but
the
the
graphs
are
pretty
much
very
similar.
B
B
In
addition,
while
we
added,
why
not
also
optimize
a
couple
of
other
things
to
make
this
easier
to
to
run
to
operate
so,
for
example,
CPU
usage
was
quite
spiky
we'll
go
into
why
that
is
mostly
because,
like
we
have
to
integrate
our
cache
every
time
and
generate
all
the
metrics,
so
CPU
load
is
very
spiky.
Maybe
we
can
optimize
for
that,
and
also
in
general
memory
would
be
nice
to
optimize
in
the
long
run.
B
So
these
are
the
three
metrics
pretty
much
that
we
want
to
look
at
along
the
way
all
right,
so
we
can
dive
right
into
it.
My
graphs
are
a
bit
off,
but
anyways.
Caching,
the
first
thing,
and
probably
the
most
helpful
thing
that
we
were
able
to
do
is
optimize.
The
caching
labor
layer
on
keep
state
metrics
in
general,
Cavanaugh's
very
nice,
as
it
gives
us
that
client
library
and
watch
API,
and
we
pretty
much
don't
have
to
take
care
of
cache
invalidation,
which
makes
this
project
a
lot
easier
in
general.
B
B
So
instead
how
about
we
get
a
new
pod?
We
generate
the
metrics
for
that
pod
and
we
will
catch
the
metrics
and
then,
whenever
Prometheus
comes
along
and
once
all
the
metrics
well
were
to
return
the
metrics
that
we
really
have
in
cash
here.
That
is
a
lot
more
optimized
that
is
in
the
best
case,
like
taking
this
memory
chunk
and
simply
shoving
it
into
the
network
card
or
the
network
interface.
That's
the
best
case.
Of
course
that's
not
what's
in
the
end
happening,
but
very
close
to
all
right.
B
So
this
is
all
under
the
premise
that
probably
more
our
kubernetes
objects
are
going
to
change.
Then
we
get
scrapes
in
so
that
kubernetes
objects
actually
changed
more
often,
and
but
a
lot
of
them
stays
the
same
while
prometheus,
regularly
scrapes
us
cool
make
this
more
specific.
Previously
we
were
keeping
the
kubernetes
object
in
cache.
Now
we
keep
the
Prometheus
object
in
cache.
B
Now.
How
much
did
this
actually
help?
This
is
on
two
hundred
fifty
notes
cluster
right
now.
A
couple
I
think
we
even
in
the
end
like
it
only
matters
how
many
objects
you
have
on
cube.
State
metrics
doesn't
really
matter
like
how
many
workloads
or
how
powerful
those
workloads
are.
Has
it
simply
like
for
enduring
text,
so
this
is
I,
think
I
think
it
and
parts
per
node.
So
quite
a
lot
of
traffic
for
cube
state
measures
itself
and
on
the
top
you
see
the
the
red
one
is
the
old
version.
B
B
We
implemented
our
own
store
and
that
uses
simply
like
metrics,
as
in
the
cache
and
we're
being
notified
of
new
objects,
we
transfer
them
into
metrics
and
then
stage
those
in
our
own
cache.
All
the
like
cash
and
validation
notifications
are
coming
from
the
reflectors.
So
not
a
lot
of
work
on
our
side
here,
cool
from
cashing
on
we
can
dive
into
compression.
B
On
the
top
here
you
have
a
snippet
of
the
Prometheus
metric
exposition
format,
and
this
is
simply
one
one
single
metric
and
you
see
here
keeps
each
secret
type
like
that
is
super
repetitive
and
as
an
engineer
you
would
say
like
okay,
let's
take
compression
and
we
save
more
than
half
of
our
network
bandwidth
in
total
in
general,
like
Prometheus,
moved
off
from
your
PC
format
into
a
text
format
with
version
2.
So
hopefully
we
can
still
optimize
on
that.
B
The
previous
cube
state
metric
version
would
gzip
all
of
this
by
default,
as
the
Prometheus
client
library
does
that
by
default.
But
we
wanted
to
play
around
with
it
a
little
bit
and
also
add
compression
into
the
new
version
of
chip.
State
metrics.
So
different
options
were
gzipping
standard,
golang,
gzipping
and
york
times
gzipping
new
york
times
is
they
wrote,
gzip
handler
library
in
the
end,
which
sounded
quite
promising,
so
we
wanted
to
try
them
all
out.
B
B
We
are
right
now,
looking
at
CPU
usage
on
the
very
top,
you
see
the
old
version
again,
we'll
go
a
little
bit
into
why
this
is
so
repetitive.
You'll
see
that
once
we
see
the
memory
graph,
but
underneath
you
see
two
green
versions
like
the
the
green
and
then
on
the
very
bottom,
a
blue
line.
There
are
two
greens:
are
the
optimized
cube
state,
matrix
version
which
gzipping
and
well
by
surprise
like
it,
does
use
more
CPU
than
the
lower
version?
Of
course,
the
low
ones
and
doesn't
do
gzipping?
B
Let's
look
in
addition.
At
memory
we
had
quite
a
big
cluster,
so
the
old
cube
state
matrix
version
were
just
simply
oom
because
it
would
well
take
away
too
much
memory.
You
see
where
it's
ramping
up
and
I'm
just
being
killed.
That
actually
explains
the
CPU
of
the
red
line
in
the
previous
graph,
but
also
now
you
see
memory
use
it
on
the
gzipping
ones
being
slightly
higher
than
noticeably
in
general
being
very
stable,
but
you
see
them
being
a
lot
higher
and
here
need
to
explain
a
bit.
B
Why
does
gzipping
take
more
memories
and
no
gzipping?
Well,
we
can't
really
keep
the
gzip
version
in
memory
as
we
have
our
cache,
like
structured
as
covenants
objects.
For
example,
if
a
new
pod
or
an
updated
pod
comes
in,
we
want
to
generate
the
metrics
only
for
that
pod
and
then
change
the
the
cache
value
for
that
pod
and
not
for
all
of
it.
If
we
would
change
it
for
all
of
it,
we
would
generate
all
the
metrics
every
single
time
over
the
entire
cache,
and
that
would
be
super
expensive.
B
So
what
we
actually
need
is
we
need
access
to
like
metrics
of
a
single
object.
That
way,
we
can't
keep
a
compressed
version
in
cache,
but
we
need
to
compress
every
single
time
so
that
way
you
used,
we
use
on
cube
state
metrics,
more
memory
on
the
gzip
a1,
but
still
like
this
could
all
still
be
perfect
for
for
networking.
So
let's
look
at
those
graphs
in
the
end,
like
scrape
durations
is
like
the
total,
like
the
network.
B
Then
you
see
on
the
very
top
the
two
green
lines
and
then
the
balloon
line.
The
green
lines
again
are
a
gzipping
enabled,
and
the
blue
lines
are
no
gzip.
Here
you
can
see
like
while
gzipping
might
have
helped
us
in
terms
of
the
network,
but
it
really
didn't
help
us
in
terms
of
CPU
time.
So
here,
like
the
blue
line,
no
gzipping
really
like
sticks
out
as
being
the
most
performant
one,
and
that
that
was
also
our
conclusion.
In
the
end,
well,
first
of
all
like
with
gzipping,
we
increase
our
size
internally
like
memory.
B
Our
memory
footprint
then
also
the
saved
improvement
on
the
network
isn't
really
not
worth
the
extra
CPU
cycles
that
we
waste
on
the
compression
itself,
especially
as
we
can't
better
compress
I'll
go
a
little
bit
into
detail.
How
we
could
improve
that
in
the
future
and
yeah
we
have
higher
CPU
utilization.
So
in
the
end,
what
do
you
need
to
keep
in
mind?
Cube
state
Metro
is
probably
very
close
to
your
prometheus,
so
that
way,
you're,
probably
on
a
very
high-performing
network.
B
B
Now,
next
to
the
caching
and
compression
we
are
in
a
very
specific
use
case.
This
actually
happened
on
Prometheus
upstream.
This
is
done
by
yarn
from
the
upstream
team.
What
we
can
do,
what
is
quite
expensive,
is
converting
floats
to
strings.
Of
course,
not
if
you
do
one,
but
if
you
do
so
many
that
you
end
up
with
fifty
megabytes
of
metrics,
you
definitely
want
some
optimizations
here
and
what
we
can
do
is
we
probably
have
default
values
like
we
probably
have
a
lot
of
zeros
and
our
metrics
a
lot
of
ones
and
our
metrics.
B
So
what
we
really
do
is
like
check
for
those
default
values
and
render
the
strings
about
easier.
That
makes
it
a
lot
faster
in
terms
of
metric
rendering,
then,
in
addition,
I'll
dive
more
into
that
in
a
bit.
But
what
is
also
very
important
if
you
want
to
be
very
specific
about
how
you
render
your
strings
in
golang,
you
might
want
to
look
into
strings
builder.
It
doesn't
give
you
so
many
options
like,
for
example,
the
FMT
package,
but
it
is
more
a
lot
more
efficient
in
the
end
like
strings
and
golang
are
immutable.
B
That
way.
Every
time
you
change
a
string,
you
need
to
reallocate
that
memory.
Instead,
what
we
can
do
is
we
can
use
byte
slices,
we
can
mute
and
that
we
can
address
a
lot
better.
So
strings
builder
simply
leverages
on
that
point
which
got
us
quite
far
all
right.
So
comparing
all
that
and
a
couple
more
versions,
a
lot,
a
couple:
more
stats
here
on
the
top,
you
see
the
old
version.
You
have
a
scrape
duration
of
22
seconds.
That
was
quite
a
bit
of
a
problem.
B
B
All
of
this
was
tested
on
a
production
cluster
actually
by
Bureau
net
SoundCloud,
which
we
are
super
thankful
for
like
to
have
some
production
workload
on
this,
then
the
next
two
versions
optimize
on
the
metric,
rendering
itself
not
on
the
caching
logic,
and
here
you
see
for
sure
and
improvement
on
scrape
durations,
but
still
we
generate
all
the
metrics
every
single
time,
Prometheus
grapes.
So
it's
not
a
huge
improvement.
Cpu
usage
got
a
lot
better.
That's
mostly
the
use
of
strings
builder
and
defaulting
on
float
conversion,
and
then
also
memory.
B
We
could
reduce
s
yeah
on
the
float,
reusing
buffers
one
and
then
also
the
loot
conversion
got
a
lot
more
efficient.
Of
course,
we
spend
less
time
in
garbage
collection,
this
reduced
memory
allocation,
but
a
lot,
so
that
was
also
very
useful
on
the
bottom.
Two,
you
see
the
caching
optimizations
here,
there's
actually
the
biggest
improvement
as
I
said
earlier.
For
our
specific
use
case,
CPU
usage
dropped
a
huge
amount,
because
simply
we
don't
need
to
regenerate
all
the
metrics
every
single
time
and
also
like
memory
dropped.
B
A
lot
as
like
caching
metrics
is
actually
a
lot
more
efficient
and
caching
kubernetes
objects
in
our
use
case
as
we're
throwing
away
most
of
the
data
anyways
and
again
garbage
collection-
hey,
of
course,
we're
not
throwing
away
all
the
metrics,
so
garbage
collector
is
really
not
that
tough
and
yeah.
We
save
a
lot
of
CPU
cycles
through
that,
as
you
saw
earlier
in
the
graphs
right.
B
All
this
resulted
in
version
1.5.
That
was
a
released
I
think
this
year,
maybe
it
January
or
something
got
good
feedback,
pretty
much,
no
bug
reports
which
I'm
bit
surprised
but
anyways
that
worked
quite
well
1.5
now
for
the
next
one.
This
is
a
community
project,
so
we
got
a
huge
amount
of
new
metrics
coming
in
all
the
time.
B
So
this
is
super
easy
and
go
to
profile
your
applications
and
wants
to
dive
into
a
couple
of
things
that
like
stuck
out
for
me
and
that
might
be
interesting
for
you
as
well,
once
you
optimize
some
go
on
kubernetes
one
is
my
profiles
were
like
runtime.
Your
slice
was
all
over
the
place
and
I
was
a
little
bit
like
okay
bro
slicing.
Maybe
we
can
optimize
for
this
and
I
looked
into
runtime
growth
slice.
B
There's
simply
the
upstream,
like
goal
line
core
function.
What
in
the
ended
us
it
grows?
The
capacity
of
the
slice
for
the
people
that
don't
write,
go
very
much
like
a
slices
is
not
an
array.
Arrays
are
fixed
types
at
compile
time
where
slices
are
variable
sized
at
runtime,
so
you
can
grow
them
in
the
end.
Slices
do
use
arrays
underneath,
but
that's
like
an
implementation
detail.
B
Okay,
so
I
thought
we're
spending
quite
a
lot
of
CPU
cycles
in
this
go
slice.
Why
are
we
doing
this?
Can't
we
like
free,
allocate
more
memory
in
advance,
can't
we
be
more
efficient
on
that
and
what
you
see
a
lot
like,
we
iterate
through
parts
and
we
iterate
through
the
properties
of
parts
or
config,
Maps
or
ingressive
and
so
on,
and
we
really
like
at
the
beginning,
we
allocate
like
an
empty
slice
of
metrics
and
I
thought.
Okay,
I'll
be
really
smart
and
well
allocate
a
certain
length.
B
Pre
allocate
all
of
it,
do
the
allocation
at
the
beginning
and
then
iterate
and
simply
insert
it
into
that
pre-allocated
slice.
We
can't
go
with
erase
because
we
don't
know
the
size
at
the
beginning.
That
would
be
super
nice.
Maybe
that
then,
could
even
stay
on
the
stack
instead
of
the
heap
but
as
at
least
like
this
is
a
good
start,
anyways
testing.
All
this
you
can
ignore
the
blue
line
on
the
bottom.
For
now
you
see
the
red
one
and
the
green
ones.
B
B
I'd
I've
to
live
a
deeper
into
the
language
itself,
and
what
you
can
really
see
here
is
like
gross
slice,
actually
grows
by
doubling
every
single
time,
pretty
much,
there's
a
little
bit
more
heuristics
around
it,
but
actually
what
we
do
is
we
don't
grow
slices
so
I'm
often
so
this
grows
put
exponentially
and
simply
like
it
doesn't
happen
that
often
so
it's
fine
and
that
way
my
optimization
didn't
really
help
a
lot.
This
is
good
to
verify
on
on
your
clusters.
B
Whenever,
like
you,
can
just
run
the
versions
next
to
each
other,
now
I
do
want
to
go
into
the
blue
line
on
the
bottom.
After
some
more
profiling,
I
saw
strings
to
slice.
Bytes
stick
out
quite
a
bit.
We
wasted
a
lot
of
CPU
cycle
cycles
on
this,
and
first
thought
for
me
was
like
bite.
Slices
come
on.
Those
are
strings
like
that's.
That's
just
on
the
type
system
level
like
how
does
that
differ
in
any
ways
and
as
I
mentioned
earlier
well
strings
are
immutable
bite
slices
or
not
to
keep
up
that
guarantee.
B
Imagine
you
have
a
string,
and
you
have
multiple
references
to
that
string
and
now
that
one
reference
changes
that
string
or
casts
it
into
bite
slice
that
way
it
has
a
mutable
reference
to
it.
It
could
mutated
by
slice
and
thus
the
other
reference
sees
a
string
which
is
being
mutated,
which
is
quite
a
problem
right.
So
in
order
to
keep
up
that
guarantee
of
strings
being
immutable,
golang
actually
takes
the
entire
memory
memory
section
and
copies
it.
So
that
was
super
heavy
on
us
and
they'll
actually
show
where
that
happened.
B
Now
this
is
actually
the
blue
line
at
the
bottom.
We
we
are
looking
at
the
CPU,
the
CPU
consumption
here,
and
that
saved
us
a
lot
for
the
1.6
version.
All
right.
All
of
this
resulted
in
the
1.6
version.
We
released
that
one
yeah,
and
that
is
the
current
version.
You
can
check
it
out
all
open
source
all
day
yeah.
It's
all
pretty
much
like
goes
down
from
one
poor
request,
so
you
can
click
through
all
of
it,
and
yeah
I
wanted
I've
loved
it
into
the
future
of
cube
state
metric.
B
Maybe
what
we
can
do
from
a
performance
side-
and
definitely
this
is
also
slide
where
I
like
this
is
just
probably
over
engineering
on
the
lower
parts
of
this
but
I'll
dive
into
it.
Anyways
first
of
all,
like
Fredrik
in
the
front
here,
is
working
on
sharding
keep
state
metrics,
don't
get
me
wrong.
That
is
not
that's,
not
the
Olfa
optimization.
This
is
actually
super
useful
on
bigger
clusters.
B
What
you
can
do
nowadays
already
you
can
chart
based
on
kerbin
edges
types,
but
if
you
have
a
lot
of
parts
and
just
one
config
map
like
sharding
on
that,
it's
not
super
useful.
So
what
we?
What
Frederick
instead
developed,
is
you
can
char
it
based
on
kubernetes,
object,
IDs
and
then
you
can
distribute
your
shorts
or
you
can
load
balance
on
your
charts
a
lot
easier,
then
in
a
in
addition,
jimmy
pointed
me
to
heap
escapes
to
look
more
into
like
when
do
stuff
the
stuff
in
cube
state
metrics,
escape
to
the
heap.
B
B
Maybe
someone
has
experience
so
I
would
very
much
appreciate
your
thoughts,
but
what
we
could
do
is
we
could,
in
the
cash,
keep
the
gzip
versions
and
somehow
like
make
use
of
community
activity
of
gzip
so
that
it's
still
performant,
even
though
we
only
G's
the
part
of
it
but
yeah
that
might
be
a
bit
tiny
bit.
Over-Engineered
yeah,
that
is
the
future
of
all
of
those
in
general.
B
Maybe
take
away,
is
huge
date
mixtures
a
lot
more
demise
nowadays,
so
we
had
a
lot
like
from
upstream
Prometheus,
but
also
inside
the
Kuban
as
community
on
this.
But
also
when
you
look
at
it
from
a
different
perspective,
as
I
said,
like
s,
the
alternative
title
from
the
talk
takeaway
is
like,
first
of
all,
monitor
your
cluster
and
application.
We
have
to
say
that
a
sec
instrumentation
here,
but
also
use
your
monitoring,
of
course,
for
alerting,
but
also
use
it
for
performance
analysis
and
make
this
available
to
your
developers,
so
they
can
debug
regressions.
B
They
can
run
their
their
software
in
different
versions
and
can
compare
it
over
long
terms.
This
was
super
helpful
for
me,
sorry
and
then
last
as
I
showed
you
like.
Not
every
optimization
is
very
helpful
and
sometimes
the
gold
memory
model
documentation
actually
gives
a
very
good
tip
on
this,
like
if
you
must
read
on
on-the-go
memory
model.
So
if
you
have
to
read
this
document
to
understand
your
code
and
understand
the
program
of
your
program,
you're
being
too
clever
and
don't
be
too
clever,
yeah,
that's
it.
Thank
you
very
much
for
you.
B
C
Thanks
for
the
duck,
it's
really
good.
My
question
is:
have
you
ever
used,
go
benchmarking
tools
because
I
can
see
you
validate
that
on
the
way
up,
reviews
which
is
fair
and
like
its
ultimate
acceptance,
which
is
good,
but
the
problem
is:
when
you
don't
have
this,
you
know
even
load
that
would
you
know
kind
of
show
test
the
thing
and
did
you
find
the
gobiins
useful?
Did
you
use
it
so.
B
What
we
do
in
cube
state
measures
actually,
at
the
very
beginning,
we
made
everything
that
cubes
eight
metrics
talks
to
abstract
it,
so
we
build
stubs
for
everything,
and
so
what
we
can
do
is
test
cube
state
metrics
with
the
gold
benchmark
tool
entirely
with
anything
around
it,
and
we
did
that
both
on,
like
smaller
scopes
or
unit
testing,
as
well
as
integration
or
kind
of
into
in
testing.
In
that
version,
all
right,
that's
that's
called
an
integration
testing.
B
So
what
we
have
now
in
our
CI
actually
is,
as
our
CI
automatically
checks
out
your
patch
and
autom
also
checks.
Our
master
runs
the
benchmarks
overall,
both
of
them
and
then
there's
a
really
cool
tool
on
this
tennard
go
that
compares
the
two
and
then
prints
it
out.
So
every
single
PR
we
can
like
check.
Does
this
introduce
any
regressions?
We
do
use
it
on
the
CI
a--?
B
We
don't
that
was
quite.
That
is
definitely
a
big
risk
that
chip
state
metrics
went
into
like
going
away
to
a
more
performant
version
of
the
client
library.
For
this
use
case,
like
I'm,
not
saying
like
the
Prometheus
plan,
library
is
not
optimized,
it
is
way
more
optimized,
but
this
is
specific
for
a
use
case.
So
this
is
a
different,
some
in
some
terms
a
different
code,
but
it
tries
to
reuse
most
of
the
concepts
that
upstream
does.
But
yes,
we're
in.
We
are
a
fork
now
in
terms
of
Prometheus
compatibility.
Yes,
this.