►
From YouTube: OPS Live w/Josh McDonald
Description
Austin Parker and Liz Fong-Jones interview Josh McDonald from LightStep at OPS Live.
B
A
B
A
C
C
A
C
C
B
A
C
B
A
C
Am
an
engineer
at
light
step
and
I've
been
here
for
some
years
before
that
I
was
at
Google.
I've
worked
on
tracing
and
logging,
and
this
was
my
first
real
foray
in
metrics.
It's
been
a
quite
exciting
time,
and
so
we're
gonna
talk
here
about
the
designs
that
we
came
up
with,
starting
with
the
original
open
census,
design
info
metrics,
taking
into
account
some
new
requirements
after
open
telemetry
started
and
then
we'll
see
where
we
landed.
Are
we
able
to
us
turn
the
pages
now
cool
so
great?
C
C
The
major
major
guideline
that
we
start
with
is
to
decouple
the
API
from
the
SDK,
and
what
this
means
to
us
is
that
we're
going
to
have
instrumentation
written
in
your
program
or
your
third-party
library,
which
links
against
only
the
API,
the
pure
API
and
then
the
operator
of
the
system,
which
will
be
somebody
who
controls
the
command
line
and
the
environment
link
options.
Or
maybe
the
main
function
of
your
program,
is
going
to
choose
and
configure
the
SDK.
C
So,
if
you're
familiar
with
open
tracing,
we
started
out
with
a
decoupling
of
the
API
from
the
SDK
in
that
world
and
didn't
have
this
SDK
concepts
the
default
SDK
concept.
So
there's
quite
a
bit.
That's
new
in
this
I
put
this
diagram
on
the
screen
here
with
diagram
was
written
for
tracing,
so
you
can
see
that,
but
it's
basically
the
same
picture
for
metrics
API
you've
got
your
user
application
and
the
third-party
libraries
you
can
get
us
the
API.
The
API
might
be
configured
with
nothing
which
case
you
get
this
minimal
implementation.
C
Sometimes
that's
called
the
no
op
implementation
and
then
it's
coupled
with
unless
it's
got
the
know
of
implementation,
your
coupled
with
an
SDK,
and
that
has
some
functionality
related
to
processing
and
data
and
then
some
related
to
protocol
that
we're
going
to
export
in
so
the
guidelines
say
that
we
should
try
to
do
some
processing
in
common
code.
We
want
to
build
a
reuse,
much
of
the
SDK
for
different
types
of
application
that
we
encounter.
C
No,
the
idea
is
that
new
exporters
can
be
written
there,
minimal
effort
and
to
this
and
we're
going
to
try
to
provide
some
extension
points
that
let
people
come
into
that
SDK
codebase
and
add
something
to
it,
perhaps
to
be
able
to
implement
new
functionality
without
rewriting
an
entire
SDK.
The
guidelines
are
also
that
we're
going
to
include
standard
open-source
exporters
for
the
open
source
systems
that
consume
this
type
of
data,
and
we
want
to
be
high
performance,
low
impact.
Those
are
sort
of
vague,
but
but
we
know
what
they
mean.
C
C
C
So
now,
since
the
API
is
independent
from
the
SDK,
what
were
what
we
came
up
with
is
that
we
needed
to
find
the
semantics
of
the
operations
and
I'm
trying
to
compare
this
with
with
this
term
of
a
facade
API,
which
is
out
there.
You
see
these
in
the
diagnostic
space
as
well.
There's
often
like
a
library
for
logging,
with
a
facade
on
it
and
the
reason
why
the
decoupling
is
not
the
same
as
creating
a
facade
API
is
that
we
need.
C
We
need
a
semantics
so
that
we
are
going
to
tell
the
SDK
what
it
means
when
we
make
one
of
these
operations
on
a
metrics
API.
Instead
of
saying
what
behavior
we
want
to
happen,
what
operations,
what
operational
behavior
we
want,
we're
going
to
say
what
it
means.
So,
the
semantics
let
the
instrumentation
convey
meaning,
and
it
also
lets
humans,
read
and
understand,
what's
happening,
and
then
the
SDKs
which
are
again
decoupled,
are
going
to
be
able
to
implement
their
own
functionality
that
are
matched
to
the
semantics
of
the
operations.
C
Well,
appropriate
this
histogram
or
any
histogram
really
is
going
to
be.
When
you
have
a
number
of
measurements,
you're
going
to
construct
different
ranges,
usually
contiguous
that
are
going
to
and
we're
gonna
count
the
number
of
measurements
that
fall
into
each
one
of
those
ranges.
So
if
you
were
measuring
latency,
for
example,
you
would
have
a
range
that
was
like
zero
to
ten
microseconds
and
you'd
have
arranged.
C
It
was
10
to
20
microseconds
and
then,
as
those
ranges
get
higher
and
higher,
you
might
make
them
broader
or
you
might
not
that's
another
one
of
these
behavioral
things
that
you
can
usually
configure
and
so
in
in
a
Prometheus
API
or
even
in
a
staff.
The
API,
you
did
have
a
method
that
said
histogram.
Here's
my
measurement
and
what
were
what
we're
trying
to
say
now
is
that
the
because
the
SDK
is
decoupled
from
the
API,
we're
not
going
to
say,
create
a
histogram
anymore,
we're
gonna,
say:
I
am
recording
a
measurement.
C
That
is
a
more
of
a
semantic
definition.
It
gives
meaning.
It
says
what
the
esier
is
a,
what
the
SDK
is
able
to
assume
about
that
event
or
what?
What
it
ought
to
be
able
to
do
with
that
information,
and
so
this
is
actually
a
pretty
pretty
significant
departure
from
the
sort
of
existing
metrics
libraries
and
I'm
gonna.
Keep
talking
about
this
theme.
C
One
of
the
reasons
why
we
don't
want
to
have
well
one
reasons
why
why
it's
important
in
a
Diagnostics
library
that
we
are
giving
semantic
definitions
and
not
real,
really
sort
of
operational
definitions
that
most
of
these
api's
are
right
only
in
the
sense
that
you
don't
get
to
read
back
anything
from
the
Diagnostics
API,
so
you're
not
reading
anything
back
you're,
not
getting
anything
from
the
SDK.
So
it's
hard
to
say
what
what
what
you,
what
you
mean
when
you're
telling
it
operationally
what
it
should
do.
I'm.
B
Actively
cleaning
from
an
API
design
perspective
right,
like
what's
happening
here
is
the
storage
format
is
not
being
specified
because
that's
a
thing
that
can
be
optimized
later,
whereas
the
upfront
thing
that
people
want
to
record
is
just
hey:
I
had
a
RPC
that
took
30
microseconds
right,
so
it's
understanding.
This
is
kind
of
the
motivation
is
not
as
kind
of
not
over
constraining
the
storage
format
at
the
time
that
measurements
are
reported
right.
C
And
and
if
you're
familiar
with
you
say,
a
Prometheus
library
you'll
know
that
there
is
a
histogram
type,
there's
a
summary
type
and
those
two
types,
those
what
we
call
value
types.
Those
are
the
output
of
the
system.
When
you
enter
this,
these
measurements,
those
are
semantically
the
same
type
and
we
are
going
to
collapse
them
into
one
metric
type.
B
C
C
Right
so
a
summary
in
these
traditional
metric
systems
are
a
summary
of
quantiles.
So
if
you
have,
if
you
consider
the
set
of
measurements
to
be
a
probability
distribution,
if
you
will
the
probability
that
the
value
is
less
than
some
say,
30,
milliseconds
or
30
microseconds
might
be
10%
and
then
so.
The
ten
percent
quantile
of
that
distribution
is
30
milliseconds
and
then
you
might
often
were
interested
in
knowing
what
the
fifteen
percent
quantile
or
the
90
percent
quantile,
the
99th
percentile
and
so
on.
99.9.
C
C
In
another
slide,
it's
there's
a
range
of
performance,
trade-offs
that
you
can
make
when
you're
computing
quantiles
and
that's
another
reason
why
I
want
to
kind
of
decouple
the
API
and
the
SDK
here
is
that
different
SDKs
are
going
to
bring
different
tricks
and
and
algorithms
to
this
to
this
space
and
what
and
what
they'll
be
able
to
do
different
things.
Sometimes
you
do
want
a
histogram,
and
sometimes
you
really
care
about
quantiles
and
the
capabilities
of
the
SDKs
are
going
to
be
different
in
that
in
that
and
in
computing.
These
this
information
and.
B
All
and
it
may
wind
up
being
especially
for
combining
metrics
and
traces
you
may
want
to
attach
the
individual
metric
measurements
to
events
that
you're
sending
out
for
the
tracing
API
right.
So
it's
kind
of
this
thing
where
the
more
flexibility
you
give
on
the
SDK
size,
the
ferret
winds
up
being,
rather
than
over,
constraining
and
pre
aggregating
right.
C
That's
a
really
good
point:
when
we
come
to
talking
about
combining
tracing
and
metrics
into
a
single
sort
of
product
experience,
though
there
will
be
definitely
a
call
for
SDKs
that
are
able
to
export
metrics
as
part
of
a
trace,
and
that
requires
a
fairly
different
type
of
aggregation,
potentially
or
maybe
you're
interested
in
exporting
raw
data.
So
again,
there's
lots
of
freedom
that
the
SDK
has
now
to
decide
what
to
do
with
these
events.
C
Let's
take
a
look
at
the
next
slide,
since
I
need
to
define
what
an
events
is
all
right,
so
I
have
given
four
sort
of
the
most
important
definitions
here
for
talking
about.
Metrics
I
want
to
say
up
front
that
the
term
metrics
itself
is
actually
dangerously
ambiguous
and
I
try
to
not
ever
use
the
word
metric
by
itself
or
metrics
by
itself.
C
So
the
term
that
I'm
using
here
to
talk
about
the
thing
program
ssin
depending
on
the
type
of
metric
instrument
that
you
have
there
will
be
different
ways
to
do
that,
and
these
are
defined
in
code.
You
declare
them
by
by
allocating
them
with
a
constructor
I'm,
going
to
example
that
on
the
next
slide
or
two
slides
from
now,
so
once
you've
allocated
these
in
your
program,
you
can
operate
on
them
and
to
report
numbers.
C
Essentially,
a
metric
event
is
when
you
actually
call
one
of
these
instrumentation
statements
that
update
symmetric
instrument,
so
the
event
will
be
I'm,
adding
to
a
counter,
for
example,
and
the
semantics
of
the
API
say
what
it
means
to
add
to
a
counter
you're
going
to
compute
a
sum
you've
added
one
new
value
to
that
sum.
For
example,
the
term
metric
labels
comes
up
very
often
one
of
the
most
powerful
things
that
we
were
able
to
do
with
labels
in
the
metric
space
also
refer
them
as
key
value
pairs,
sort
of
generate
generically.
C
So
labels
are
one
of
the
major
features
of
the
API
and
you
can
attach
them
to
these
events
and
we'll
get
to
talking
about
the
ways
that
you
do
that
later.
The
word
aggregation
is
hard
to
avoid
here
as
well.
It's
pretty
important
to
understand
that
metrics
are
the
metrics.
Events
are
quite
expensive
if
you
were
to
record
them
individually
and
literally,
send
them
out
over
the
wire
it
wants
1.1
to
1.
C
So
very
often
we
make
an
effort
to
optimize
the
performance
of
the
metrics
API
by
aggregation,
and
there
are
two
types
of
aggregation
that
we
will
end
up
talking
about
the
first
and
most
common
is
temporal.
So
let's
say
you
have
a
minute
worth
of
data
and
you're
aggregating
a
counter.
What
you're
gonna
do
is
you're
gonna
compute.
That
sum
over
one
minute,
let's
say
and
then
export
just
one
value
at
the
end
of
the
minute.
This
lets
you
export
a
single,
a
single
value,
instead
of
all
the
events
that
were
contributing
to
that
sum.
C
When
it
comes
to
talking
about
labels,
once
again,
we
may
be
interested
in
aggregation
over
label
values,
and
so
the
term
spatial
aggregation
comes
up.
We
think
of
these
labels
as
additional
dimensions
on
the
data,
and
so
the
the
term.
Spatial
aggregation
applies
because
each
value
for
a
dimension
or
Lakey
value
is
a
sub
part
of
that
dimensional
space
and
when
we
do
spatial
aggregation,
we're
dropping
dimensions
in
order
to
collapse
data
into
a
smaller
space.
C
B
C
Right
and
this
can
be
done
either
in
the
process,
you're
exporting
data-
and
you
might
ask
why
would
you
put
on
values
on
a
metric
that
you're
not
going
to
export
we'll
get
to
that
a
little
bit
later?
But
definitely
in
the
case,
when
you're
aggregating
data
from
multiple
machines,
if
the
machine
is
or
the
host
is
one
of
the
attributes
that
you
have,
you
might
end
up
wanting
to
drop
that
in
order
to
look
at
a
single
value.
C
Okay,
let's
see
the
next
slide
here,
talks
about
the
three
kinds
of
instrument
that
we
have
again.
These
are
the
semantic
kinds
each
one
of
these
has
a
different
meaning
and
each
one
of
these
supports
just
exactly
one
operation.
So
the
ones
the
two
that
are
most
familiar
to
you
if
you've
dealt
with
many
of
the
existing
metrics
api's
are
going
to
be
counter
engage
counter
is
has
an
operation
named
ad
and
usually
the
the
meaning
of
this
operation?
C
Is
that
you're
going
to
contribute
something
to
a
sum
or
a
rate,
and
that
is
fairly
actually
I'd,
say
the
most
straightforward
of
these
metric
instruments?
You're
computing,
a
sum
and
you're
gonna,
add
values
to
it.
You
may
add
one
value
or
you
might
add,
you
might
add
one
value,
that's
10
or
you
might
add,
10
values
that
are
1.
It
doesn't
matter
because
the
semantics
are,
you
need
a
sum,
and
so
counters
are
the
easiest
per
tip
to
to
aggregate
temporarily,
especially
because
they
you
can
do
them
with
a
simple
atomic
operation.
C
If
you
have
a
counter
in
memory-
and
you
want
to
add
to
it,
you
just
add
and
that's
it.
The
gauge
instrument
is
a
little
bit
more
difficult
to
understand,
usually
you're,
using
a
gauge
when
you
want
to
have
a
current
value.
You
might
have
a
current
value
because
there's
some
complicated
piece
of
code
that
you
aren't
responsible
for
that
computes
it
for
you.
B
The
more
prototypical
case
of
that
is
something
that's
even
like
completely
outside
of
your
control
write,
something
like
the
CPU
utilization
on
your
host
right,
where
it's
has
a
current
value.
You
can
look
up
the
current
value,
but
it's
not
necessarily
something
that
gets
changed
by
your
code
right.
C
So
load
measurements,
for
example,
that's
exactly
the
most
canonical
example,
I
would
say:
CPU
load
and
the
final
type
measure
that
the
semantic
type
here
is
called
measure.
So
we
call
them
measure
metrics
or
measure
metric
instruments.
It
has
an
operation
called
record
and
the
the
semantics
are
that
you're
entering
an
individual
measurement
and
that's,
unlike
the
other
two,
where
the
number
of
calls
to
counter
add
or
the
number
of
calls
to
gauge
set
are
not
important.
C
Quantiles
summary
metrics
summary
values,
as
we
called
them
earlier
or
histograms,
and
these
are
the
ones
that
can
get
expensive
because
you
might
need
to
actually
record
every
raw
value
in
order
to
compute
an
exact
exact
quantile,
for
example,
and
we're
gonna
talk
later
about
how
there
are
ways
to
compute
estimated
quantiles
that
are
significantly
cheaper
and
there's
certain
like
in
like
very
cheap
operations.
You
can
compute
like
Max
and
min
that
don't
require
much
at
all,
but
that's
much
less
information.
B
We
have
to
stop
and
reinforce
here
really
quickly
like
the
difference
between
counter
and
measure
again
for
a
bed,
a
counter
might
say
you
know,
might
measure
how
many
requests
per
second
am
I.
Getting
by
measuring
had
the
cumulative
sum
of
how
many
requests
I've
received
since
the
process
started,
whereas
the
measure
actually
takes
an
additional
attribute
right.
Not
just
you
know,
hame
inferring,
the
number
of
requests
by
one,
but
instead
actually
saying
you
know,
hey
in
this
request,
took
30
milliseconds
or
30
microseconds
to
compute
right.
B
C
The
way
I
think
of
it
also
is
that
the
number
the
number
of
calls
matters
in
this
case,
so
you
you're
interested
in
the
count,
whereas
the
number
of
times
you
called
add,
doesn't
matter
that
all
that
matters
is
a
sum.
But
if
you
call
record
you're
not
only
giving
a
value,
you're
saying
one
event
happened,
because
you
might
be
interested
in
that
the
rate
of
events
independently
from
the
distribution
of
quantile
information,
for
example.
C
Now,
let's
see
what's,
let's
move
on
to
the
next
slide?
There's
I
have
an
example
here
of
how
you
allocate
an
instrument,
so
we've
talked
about
the
three
instrument
types
or
the
three
kinds
of
semantics
instrument
here
each
kind
supports
integer
and
floating-point
instruments,
so
that
gives
us
basically
six
constructors
to
begin
with.
I've
shown
this
example
here
and
I
want
to
declare
that
I
am
skipping
over
a
very
tricky
part
of
this
whole
open,
telemetry,
API
SDK
discussion,
which
is
how
you
deal
with
global
variables
or
program
initialization.
C
C
Not
sure
I'll
discuss
it
right
now.
The
idea
of
with
keys
is
that
now
we've
decoupled,
the
API
and
the
SDK,
so
we're
not
telling
the
SDK
exactly
what
to
do
or
what
operational
behavior
should
have.
But
when
you
do
declare
one
of
these
instruments,
the
SDK
is
going
to
want
to
know
how
to
deal
with
it
and
very
often
we
need
to
have
aggregation
in
the
process
in
order
to
get
good
performance.
C
So
if
the
SDK
has
no
other
configuration,
if
it's
a
fairly
simple
SDK
or
you
have
haven't
given
any
of
any
configuration
with
keys
here,
serves
as
an
advisory
way
for
the
programmer
to
say
what
you
should
do
with
this
metric.
So
given
no
other
information,
the
good
default
here
is
probably
to
take
the
keys
named
project
and
protocol
in
this
case
and
use
those
two
dimensions
to
pre
aggregate
the
data.
C
Again,
though,
this
is
just
an
advisory
recall,
and
that's
because
we've
decoupled
the
API,
the
these
are
good
defaults,
but
the
SDK
is
is
still
free
to
do
anything
it
wants.
The
SDK
could
override
this
configuration
or
it
could
do
anything
it
wants
duh.
Okay,
let's
continue
now.
The
next
slide
talks
about
optional
restrictions.
We
have
just
described
exactly
three
types
of
metric
instrument,
a
counter
gauge
and
measure
types,
try
to
call
them
kind,
soo.
The
way
to
word
type.
Actually.
Sorry
and
each
of
these
has
a
default
behavior.
C
When
we
looked
at
the
existing
systems
and
the
types
of
use
that
you
have
for
these
instruments,
we
found
that
there
are
definitely
in
each
case,
one
fairly
common,
but
less
less
common
alternate
behavior.
So
let's
talk
about
counter.
For
example,
the
default
behavior
for
a
counter
is
monotonic,
and
that
means
that
the
values
that
you
add
to
it
have
to
be
positive
or
zero
non
negative,
and
this
is
useful
because
very
often
with
counters,
we
want
to
compute
a
rate
and,
if
you're,
counting
negative
values,
it
defeats
the
the
value
of
a
rate
calculation.
C
So
if
you
declare
a
counter,
it's
default,
behavior
is
to
be
monotonic.
Is
it
will
the
the
SDK
should
reject
a
negative
value
and
you're
able
then
compute
a
rate?
The
alternate
behavior
in
this
case
is
non
monotonic.
If
you
want
to
compute
account,
but
you're,
counting,
positive
and
negative
values,
then
you
would
declare
the
alternate
behavior
by
saying
I
want
a
non
monotonic
counter.
The
SDK
will
then
accept
your
negative
values
and
will
allow
you
to
have
a
sum
that
rises
and
falls.
C
One
of
the
canonical
examples
of
this
is
you'd
like
to
count
how
many
bites
a
particular
data
structure
is
using
and
every
time
you
allocate
you're
gonna
count
positive
value,
and
every
time
you
do
Cait
you're
going
to
count
a
negative
value.
This
lets
you
compute
sums,
but
you
can't
compute
rates
using
a
non
monotonic
counter.
In
the
case
of
a
gauge,
the
the
default
is
non
monotonic.
This
is
a
little
bit
of
a
tricky
definition.
Non
monotonic
gauge
just
means
you
can
give
it
any
value.
C
You
want
the
we
use
this
monotonic
gauge
to
describe
a
gauge
that
can
arise.
So
when
you
set
the
next
value,
it
should
be
higher
than
greater
than
the
last
greater
than
or
equal
to
the
last
value
that
you
set.
This
turns
out
to
be
fairly
important
for
the
Prometheus
system,
which
which
has
traditionally
dealt
with
gages
monotonic
gauges
as
a
way
to
reflect
a
counter,
and
so
we've
preserved
this
behavior.
C
If
you
have
a
value
that
is
exported,
but
is
not
something
where
you're
again
able
to
you're
not
actually
receiving
individual
events,
it's
one,
that's
another
thing
that
you're
observing
like
CPU
load.
Let's
talk
about
CPU
usage:
this
is
a
value
that
only
rises.
You
don't
get
to
compute
it,
but
you
can
read
it
once
in
a
while.
So
you
might
have
a
gauge
that
says:
CPU
usage.
It
will
be
monotonic
because
your
CPU
usage
never
falls
over
the
life
of
process.
So
that's
a
good
example
of
a
monotonic
gauge.
C
So,
lastly,
the
measure
instrument
has
default
behavior
of
absolute.
That
means
that
by
default
we
expect
values
are
going
to
be
greater
than
or
equal
to
zero.
This
is
standard
behavior,
it's
probably
the
more
common
that
we
found
out
there.
One
of
the
advantages
there
is
that
you
can
use
a
measure
to
compute
a
rate
again
just
like
a
counter.
If
all
your
values
are
positive,
this
allows
you
to
compute
the
count
and
a
sum
which
computes
a
rate
at
the
same
time.
C
So
you
might
be
interested
in
knowing
how
many
requests
were
there
and
what
was
the
rate
of
bites?
Let's
say
so.
If
you're
recording
the
number
of
bites
per
request,
you
could
calculate
the
rate
of
requests
and
the
rate
of
bites.
Now
the
non
absolute
measure
is
provided
for
the
opposite
case.
Where
you
don't
have
to
require
positive
or
negative
zero
values,
you
might
have
negative
values.
These
are
cases
where
I
don't
know.
C
You're
measuring
temperature
in
Celsius
and
the
values
are
simply
negative,
so
you'd
like
to
have
a
measure
that
supports
negative
values,
all
right.
Let's
go
on
to
the
next
slide
now
here
we're
gonna
start
to
talk
about
performance,
we're
still
talking
about
the
API,
but
a
number
of
factors
related
to
performance,
I've
entered
into
the
API,
we're
gonna,
try
and
talk
about
those
next,
so
we've
talked
about
temporal
aggregation
now
a
few
times.
This
is
the
primary
tool
for
controlling
metrics
costs
and
it's
fairly
simple
in
the
case
of
a
counter
or
a
gauge.
C
If
you
have
a
counter-
and
you
have
a
number
of
count,
events
or
add
events-
that
is,
you
add
them
into
a
current
sum-
you're
only
going
to
need
to
export
the
current
sum
once
in
a
while,
instead
of
exporting
all
the
individual
counts
with
a
gauge
is
similar,
you
can
record
the
last
value
of
a
gauge
and
export
just
that.
So
if
you
have
100
calls
to
gauge
a
set-
and
you
want
to
export
the
current
value,
you
don't
need
to
export
a
hundred
values.
C
You
can
only
export
the
last
value,
that's
the
semantics
of
the
gauge
instrument.
So
there
are
two
major
cost
concerns
that
come
up
when
we
deal
with
metrics
api's.
The
first
is
labels.
Metrics
labels
are
going
to
be
expensive.
We'll
talk
a
bit
about
more
about
this,
but
the
intention
of
having
labels
is
that
we're
going
to
aggregate
them
independently.
So
if
you
have
high
cardinality
in
your
label
set,
you
may
end
up
occupying
a
lot
of
memory
to
store
individual
counts.
C
B
For
instance,
if
you
had
individual
user
IDs,
even
if
one
user
hits
you
once
suddenly,
you
have
to
create
and
track
the
label
set,
even
if
the
values
are
all
0
over
and
you
would
have
to
keep
that
I've
infinitum
until
your
process
research.
So
that's
kind
of
the
reason
why
we
say
that
high
cardinality
metrics
are
very
difficult
to
achieve.
Well,.
C
C
You
may
actually
have
to
compute
that
from
all
the
data,
so
you
could
have
to
record
all
the
data
and
then
compute
quantiles
or
you
might
have
to
compute
a
fairly
expensive
algorithm
that
approximates
or
exactly
it.
Computes
these
these
on
the
fly
using
a
stream
algorithm,
so
measure
instruments
are
generally
going
to
be
expensive,
no
matter
what
you
do
and
we'll
deal
we'll
look
at
some
ways
to
control
those
costs
as
well.
Ok,
let's
move
on
there.
The
next
slide
is
talking
about
calling
conventions.
These
are
some
of
the
ways
that
we
address
cost.
C
So
if
you're
familiar
with
existing
metrics
libraries,
you
you
know
about
the
Prometheus
and
you
may
know
about
stats
D.
The
theme
here
is
that
we
keep
cut
that
we
kept
coming
into
as
we
design
as
this
API
is
that
the
users
who
come
from
prometheus
are
going
to
expect
the
certain
calling
pattern
that
they've
become
familiar
with
and
for
that
they
get
pretty
good
performance,
but
they
may
be
restricted
in
terms
of
labels.
C
This
bound
instrument
calling
convention
is
one
as
much
like
Prometheus
you
have
an
instrument
you're
going
to
bind
it
to
a
set
of
labels
ahead
of
time.
This
gives
you
a
bound
instrument
which
you
can
then
call
quickly
to
say
a
door
set
a
record
on,
so
you
pre
declare
your
labels,
and
then
you
get
this.
This
thing
that
you
can
use
to
very
quickly
record
a
metric
or
very
quickly
add
to
a
sum
or
very
quickly,
add
to
a
gauge
you've
done
a
lot
of
pre
computation
in
order
to
make
this
fast.
C
If
you
come
from
the
other
side
of
the
world,
stats
D,
for
example-
or
if
you
have
a
case-
that's
just
not
something
you're
going
to
reuse,
it
might
be
a
status
code,
that's
just
sort
of
like
random.
You
don't
know
that
you're
going
to
reuse
it.
It
happens
in
the
moment,
you're
just
going
to
make
a
single
call.
So
this
is
called
the
direct
calling
convention.
In
this
case
you
just
have
the
label
set,
you
have
an
instrument
and
you
have
a
value.
You're
gonna
call
it
once.
C
This
is
much
more
like
a
stats
to
the
instrument.
These
two
modes
are
the
sort
of
primary
performance
control
that
you
have.
If
you
can
pre-compute
your
labels
and
pre-allocate
your
bound
instruments
and
that'll
be
much
faster
for
you,
but
there
are
cases
where
you
can't
roll,
or
there
are
cases
where
it's
simply
not
convened
to
do
so.
So
that's
the
case
when
you
end
up
using
direct
calls,
we
have
a
third
calling
convention
called
batch.
C
You
call
batch
record
on
the
on
the
meter
with
a
set
of
labels
and
a
set
of
instruments
and
measurements
or
values.
That
is
so
you
can
say,
I.
Have
this
label
set
I'm
going
to
update
five
metric
instruments?
All
at
once
got
some
examples
of
these
on
the
next
few
slides.
So
let's
talk
first
on
the
next
slide
about
reusable
label
sets.
It
turns
out
that
the
most
expensive
aspect
of
dealing
with
metrics
is
simply
constructing
and
manipulating
sets
of
labels.
C
So,
if
you're
going
to
do
aggregation
by
a
set
of
labels
very
often
inside
the
SDK
need
to
look
up
something
based
on
that
set
of
labels
and
labels
are
going
to
be
either
a
list
or
a
map,
some
ketta
value,
and
very
often
in
a
programming
language.
You
cannot
use
a
list
or
a
map
as
a
key
in
another
map,
so
there's
some
computation
that
we
need
to
do
in
order
to
get
from
our
set
of
labels
to
a
thing.
That's
like
an
entry
in
my
SDK.
C
That's
going
to
contain
temporal
aggregation
or
for
that
particular
entry,
and
this
turns
out
to
be
the
highest
cost
of
dealing
with
metrics,
whether
you're
using
the
bound
instrument
calling
convention
or
the
direct
calling
convention
or
the
record
batch
calling
convention
it's
most
expensive
to
construct
the
labels
in
the
first
place.
So
it's.
B
C
B
C
Up
using
that
again
again,
but
at
least
you're
handling
handing
this
to
the
SDK
and
if
you
do
end
up
using
it
again
and
again,
the
SDK
is
able
to
perform
a
single
computation,
for
example,
to
encode
a
string
representation,
maybe
that
it
can
then
use
every
time
you
use
that
set
of
labels.
So
this
optimization
applies
to
all
the
calling
conventions
you
can
use
label
sets
in
every
one
of
those
cases,
and
this
makes
it
possible
for
an
application
to
lower
the
cost
of
metrics
by
planning
ahead.
C
Essentially,
this
it
turns
out
to
be
a
fairly
significant
optimization
if
you're,
using
the
bound
instrument
calling
convention-
it's
not
so
again,
so
so
important.
Let's
turn
the
next
slide
and
see
an
example
in
the
bound
instrument
call
you
have
a
set
of
labels
that
we
allocated
using
the
call
in
the
previous
slide.
You
have
a
bytes
index
count
instrument
that
you've
allocated
and
in
the
example
here
you
say,
bytes
indexed
count
bind
you
hand
at
the
label
set.
This
gives
you
a
bytes
indexed
count
bound
instrument
for
this
request,
because
it's
this
up
we're.
C
It
was
still
expensive
to
compute
that
labels
once
but
at
least
the
set
of
labels
once
but
at
least
you're
going
to
amortize
that
cost
using
advice
about
instrument
I
confess
that
this
is
a
little
bit
of
a
contrived
example,
because
you
wouldn't
perhaps
need
to
call
this
Rik
bytes
indexed
add
on
every
loop
iteration.
You
could
compute
a
total
and
add
it
once
because
it's
a
counter
in
the
example,
but
if
you're
using
a
measure
instrument,
this
would
be
pretty
significant,
so
you'd
say
record
the
value
of.
C
If
you
wanted
to
compute
a
distribution
or
a
histogram
or
a
summary
of
the
size
of
bytes
indexed.
You
would
then
use
a
measure
instrument
and
you
would
record
each
size
at
the
end
of
the
loop.
You
would
you
would
be
done
and
you
would
only
serialize
that
label
set
once
so.
That's
the
benefit
of
using
the
bound
instrument,
so
this
would
be
a
very
fast
call
in
this
case.
C
Sometimes
it's
not
convenient
to
use.
Let's
look
at
the
next
slide.
Sometimes
it's
just
not
convenient
or
to
to
use
one
of
these
in
sound
instruments,
because
maybe
you
don't
have
anywhere
in
your
program
to
store
that
state
or
it's
deep
in
some
library
call
where
you
can't
make
that
optimization.
For
example,
in
this
case,
you're
going
to
simply
call
it
directly
so
I've
I've
summed
the
value
of
this
size.
Computer
total
and
now
I'm
calling
bytes
index
count.
Add
I've
got
a
passing
I'm
using
a
golang
example
here,
so
we've
got
a
context
object.
C
B
C
And
I
have
a
slide
about
that
later.
If
you
were
not
dealing
with
going,
you
would
pass
an
open,
telemetry
context.
The
Guerlain
context
hides
that
for
you,
but
it's
the
same
idea
in
the
go
in
the
context
are
going
to
be
distributed.
Talk
about
that
a
little
bit
later
later,
so
I
have
in
the
next
slide.
C
I
had
one
more
example
of
the
batch
calling
convention
just
to
make
this
the
three
calling
conventions
clear:
if
you
have
a
single
label
set
and
several
measurements
that
you
want
to
update
you
can
out,
you
can
update
all
the
instruments
at
once
with
a
single
record
batch
call.
This
may
be
faster,
for
example,
if
the
SDK
is
gonna,
take
a
lock
in
response
to
a
record
batch.
Then
then
you
might
be
better
off
doing
this
once
instead
of
once
per
instrument.
B
To
use
the
example
of
when
people
might
use
the
record
batch
rate,
like
often,
if
you're
writing
an
RPC
handler,
you
might
want
to
record
the
you
know:
number
of
bytes
indexed,
the
number
of
milliseconds
the
request
took.
You
know
the
increment,
the
success
counter
increments.
The
total
number
of
requests
received
do
not
increment
the
errors
counter
and
so
forth,
and
those
almost
here
are
the
same
labels.
So
it's
quite.
C
C
Right
and
that's
a
very
common
example
actually
is,
at
the
end
of
a
request
handler
you're
going
to
update
7
or
17
metrics
and
you've
got
one
label
set.
This
is
again
an
example
where
the
the
optimization
of
computing
a
label
set,
may
not
be
super
important
because
you're
going
to
compute
that
label
set
once
and
use
it
for
all
those
metrics.
You
could
and
it
may
not
be
much
more
efficient
than
calling
17
metrics
updates
all
at
once.
But
this
is
a
also
a
convenient
calling
pattern
for
your
code.
C
You
don't
have
to
call
17
times
and
so
on.
Alright,
let's
look
at
the
next
slide.
We
talked
about
label
sets
a
little
bit
further
I
mentioned
earlier
that
it.
If
you're
going
to
export
some
data
and
you're
going
to
perform
a
spatial
aggregation,
why
would
you
bother
putting
those
labels
into
the
label
sets
in
the
first
place?
Well,
one
answer
is
that
the
SDK
might
might
override
your
configuration
and
decide
to
export
all
the
labels,
and
this
is
pretty
common
in
in
certain
stats.
C
The
usages
so
just
apply
any
label
you
want,
because
there
are
good
reasons
to
do
that.
You,
you
can
record
that
information,
and
then
you
have
a
lot
of
you
have
the
ability
to
go
backwards
in
time
and
look
at
labels
that
you
never
thought
you
might
use
by
looking
through
your
old
metrics
data,
so
the
default
SDK
provides
this
recommended
keys.
This
is
that
advisory
thing.
We're
talking
about
that
lets
the
programmer
to
declare
what
keys
they're
interested
in
by
default
and
the
SDK
is
able
to
do
whatever
it
wants.
C
So
when
the
programmer
puts
more
labels
into
the
label
set
that
are
actually
being
aggregated,
whether
it's
because
recommended
keys
are
that
are
being
taken
or
whether,
because
the
SDK
has
decided
for
its
own
reasons,
which
labels
to
aggregate
by
you
would
end
up
having
what
we
call
additional
labels.
The
additional
labels
are
there
and
they
get
spatially
aggregated
away.
When
you
do
export,
so
you
may
ask:
why
do
we
want
them
in
the
first
place?
And
the
answer
here
is
that
SDKs
can
explore
a
lot
more
data
than
just
the
preoccupation.
C
So
one
of
the
common
applications
is
to
export
samples
you'd
like
to
sometimes
these
are
called
exemplars,
so
you
can
export
in
addition
to
the
pre
aggregated
data,
you
can
export
sample
data.
That
might
be
individual
points
in
that
metric
update
to
say
not
just
the
dimensions
that
were
aggregated
but
all
the
dimensions
that
were
present,
and
this
might
be
helpful
for
debugging
or
like
looking
deeper
into
the
data
since.
B
C
So
if
you
do
have,
let's
say:
I
user
ID
and
you're
not
going
to
include
that
user
ID
in
the
aggregation,
but
you
know
that
you
had
a
hundred
user
IDs
in
this
minute.
Maybe
you
can
arrange
to
compute
one
trace
for
every
user
ID
without
exporting
the
user
IDs
in
the
metrics
stream.
So
this
additional
information
provides
ways
that
the
tracing
and
the
metrics
SDKs
can
work
together
to
export
more
information.
C
Alright
hope
that
made
sense.
Let's
move
on
to
the
next
slide.
This
is
the
one
that
you
asked
about
earlier.
It
is
the
context
argument
that's
passed
to
every
one
of
these
calling
conventions
does
contain
the
open,
tama
tree
context,
and
this
includes
distributed
context
or
what
we
call
correlation
context.
These
are
label
values
and
key
values
that
came
from
parents
in
a
RPC
hierarchy,
for
example,
or
the
parents
in
the
trace
that
were
passed
down
and
propagated
through
through
the
through
calls
to
the
current
span.
C
Let's
say
that
are
available,
so
the
default
SDK
does
not
automatically
use
this
information.
It's
certainly
something
that
a
very
smart
SDK
could
use.
There's
a
lot
of
cost
concern
here,
so
we've
taken
the
fairly
simple
approach
is
saying
that
that
the
default
SDK
will
not
automatically
apply
distributed
correlation
context,
but
the
user
is
able
to
extract
that
information.
I
haven't
included
an
example
of
it
here,
but
the
correlation
context
API.
C
It
provides
a
way
to
ask:
what's
the
value
of
this
label
in
the
distributed
context,
and
you
can
then
put
those
into
a
label
set
and
use
them
on
your
metrics,
for
example.
This
would
be
an
example
of
the
one
of
those
sort
of
early
use
cases
for
context
prop
distributed
context.
Propagation,
including
correlations,
was
exactly
for
this
to
be
able
to
do
real-time
aggregation
in
the
children
of
your
trace
by
some
value
that
was
provided
by
the
parent.
C
So,
let's
suppose
that
you
want
to
count
the
number
of
bytes
that
are
being
read
on
a
disc
at
the
very
bottom
of
your
RPC
hierarchy,
and
you
want
to
count
them
by
the
user.
That's
making
the
request
all
the
way
at
the
top
of
your
hierarchy.
What
you'll
do
down
at
that
disk
is
extract
the
user
from
the
correlation
context
and
apply
it
to
your
metric
updates
manually.
C
It's
certainly
possible
that
an
SDK
could
do
this
automatically
and
you
can
imagine
all
kinds
of
ways
to
figure
this
there's
definitely
some
academic
work
that
talks
about
how
how
to
do
this
in
a
sort
of
clever
and
automatic
way.
But
we've
left
that
out
for
now
you
can
do
it
automatic
you
can
do
it
manually,
okay,
we're
sort
of
done
talking
about
the
API
and
all
the
considerations
that
we
put
in
there
for
cost
control.
B
C
Earlier,
on
the
very
beginning,
this
slide,
the
library
guidelines
said
that
we
should
provide
extension
points
and
we
should
make
these
exporters
fairly
easier
right.
I'm,
just
gonna
run
through
this
really
quickly,
so
the
SDK
that
we've
got
so
far
has
this
thing
called
a
mute
meter.
Its
job
is
to
just
sort
of
keep
track
of
all
the
things
that
are
in
flight
and
that,
basically,
is
the
pairing
of
an
instrument
and
a
label
set
in
any
active
state
that
hasn't
been
exported.
C
This
is
the
sort
of
front
line
between
the
API
and
the
SDK,
and
this
is
where
you
really
want
to
focus
on
high
concurrency,
so
try
to
avoid
locking
as
much
as
possible
or
whole
blocks.
Very
briefly
after
we
pass
through
the
SDK.
There
is
this
concept
of
an
aggregator.
This
is
the
thing
that
implements
some
type
of
aggregation
and
I've
talked
about
several
types
of
aggregation
so
far
and
there's
another
slide
to
talk
about
specifics
that
we
have
there's
this
concept
of
an
aggregation
selector.
C
This
is
a
policy
that
is
depends
on
the
capabilities
of
the
exporter
and
the
desires
of
the
system
or
the
user.
Configuration
that
says
what
type
of
aggregation
do
you
want?
What's
your
tolerance
for
cost,
if
you
want
very
precise
quantile
information,
you'll
choose
a
different
aggregation
and
if
you
want
very
cheap
quantile
information,
like
only
the
mean
of
the
Macs
you'll,
choose
a
very
cheap
aggregation.
So
this
is
a
the
point
where
we
configure
the
policy
about
which
aggregations
to
use.
C
Okay
next
slide,
there's
a
concept
of
a
batcher
I'm,
not
sure
this
is
the
best
name
to
be
honest,
but
with
this
this
type
does
this
this
part
of
the
SDK
does
is
decide
about
spatial
aggregation
which
labels
are
we
going
to
export
its
job
is
to
reduce
the
dimensionality
by
applying
spatial
aggregation
and
then
it
what
it
does.
Is
it
computes
a
set
of
check
points,
each
independent
metric
instrument
and
label
set
that
will
be
exported
as
checkpoint,
that's
computed
by
this
batcher,
and
so
there's
policy
here
as
well.
C
That
says
which,
which
labels
do
you
want
to
aggregate
by
and
combined
with
which
aggregations
that'll
provide
all
the
information
to
an
exporter.
The
concept
of
a
checkpoint
set
is
just
for
every
active
instrument
and
all
the
label
sets
that
we're
exporting
by
you
have
a
checkpoint
of
the
aggregator
and
the
aggregator
defines
its
own
data
model,
because
different
metric
instruments
have
different
data
models,
so
the
aggregator
for
a
counter
is
different
from
the
aggregator
for
a
gage
and
that's
different
from
the
aggregator
for
a
measure.
C
The
concept
of
a
controller
is
really
just
deciding
when
to
perform
this
checkpoint.
If
you're
doing
push
based
metrics
reporting,
you
might
have
a
timer
set
up
that
says:
I'm
going
to
export
data
every
30
seconds,
but
some
systems
do
pull
based
metrics.
So
then,
you're
gonna
have
some
sort
of
service
that
responds
to
a
request
and
decides
when
to
checkpoint
metrics.
C
Each
of
these
is
separate,
separated
in
the
SDK
so
that
you
can
swap
in
a
different
object
due
to
do
different
behavior.
Lastly,
we
have
the
exporter.
This
is
all
its
job,
then
is
to
do,
is
to
take
a
checkpoint
set,
which
was
computed
from
the
set
of
aggregators
and
the
batcher,
and
the
check
points
and
just
formats
it
and
sends
it
off.
C
C
B
C
Exactly
all
right,
we're
almost
finished
here,
I
want
to
talk
a
little
bit
on
the
next
slide
about
how
we
aggregate
measure
instruments.
I've
listed
four
on
this
slide
measure
instruments
again
are
the
ones
that
tend
to
be
expensive
because
to
compute
quantiles
you
need
all
the
data
and,
if
you're,
if
you're,
using
less
than
all
the
data
or
reporting
less
than
all
the
data,
you
must
be
approximating
in
some
way.
So
because
there's
a
tremendous
range
of
costs
here,
we
have
offer
we
offer
the
default.
C
Sdk
is
going
to
offer
several
options
the
cheapest
one
that
we
have
is
called
min
max
some
count.
The
reason
why
I'm
in
max
some
count
is
so
cheap
is
that
it
requires
exactly
4
words
of
memory
per
aggregator.
It's
Adickes
low
of
one
memory
to
compute
the
mean
the
max
of
some
an
account.
You
don't
have
to
remember
all
the
values
you
just
need
to
compute
a
little
bit
of
information,
and
this
is
actually
often
all
you
need,
and
if
I
was
to
choose
a
default,
this
would
probably
be
my
choice.
C
I
think
people
should
probably
opt
into
expensive
metrics
when
they
want
them
and
not
have
that
when
they
don't
so
now.
The
next
one
is
the
idea
of
a
sketch.
This
is
an
approximate
algorithm
that
we
can
use
to
estimate
selected
quantiles.
So
there
are
a
number
of
algorithms
out
there
for
doing
this.
In
the
go
implementation,
we've
included
the
data
of
DD
sketch
algorithm.
That's
pretty
recent.
There
are
other
algorithms.
You
may
have
heard
of
t
digest,
Q
digest,
gk
sketch,
there's
a
lots
of
them
out
there,
and
one
of
these
will
be
included.
C
The
idea
is
so
that
you
can
have
a
sort
of
compromise
between
very
cheap,
which
is
just
the
minimax,
some
account
or
an
exact
report,
which
would
include
very
precise
information
about
quantiles
or
even
just
raw
information
that
would
be
recording
all
the
all
the
measurements
exactly
so.
Lastly,
there's
this
histogram
and
that's
a
very
specific
output.
If
you
want
a
histogram,
you
should
ask
for
it
very
often
if
you're
computing
histogram,
you
are
interested
in
some
exact
viewing
range
that
you
need
and
maybe
there's
a
reason
why
you
want
that
for
your
system.
C
B
Off
I
think
between
histograms
and
percentiles,
where,
if
your
histogram
bin
ranges
are
wrong,
then
your
percentile
measurement
went
up
in
the
middle
of
a
bin
and
you
have
no
idea
kind
of
where
and
that,
then
it
falls
and
you
have
to
approximate.
So
these
are
kind
of
all
various
ways
of
kind
of
grasping
at
the
elephant
d'être
to
figure
out
like
what
shape
does
it
have
right.
C
If
you
know
the
range
in
the
distribution
really
well,
you
might
get
away
with
a
histogram
and
it's
gonna
be
very
accurate
for
you.
You
can
compute
quantiles
from
histograms,
but
there's
a
chance
that
you're
not
going
to
get
very
much
useful
information
from
that.
So
my
guess
is
that
the
an
application
is
gonna
use
a
sketch
or
when
these
knickknacks
counts
and
then,
if
you
specifically
need
a
histogram,
maybe
that's
what
you'll
get
histograms
are
often
configured
with
either
linear
buckets
or
exponential.
Longer
sorry,
logarithmic.
B
C
Or
even
variable
buckets,
which
can
be
maybe
the
output
of
something,
but
but
in
the
terms
of
an
aggregator,
you
need
to
know
upfront
what
your
histogram
are
otherwise
you're
doing
something
more
approximate
like
a
sketch,
there's
a
there's,
an
algorithm
or
a
data
structure
out
there
called
HDR
histogram
it's
implemented
in
Java,
it's
a
it's
been
it's
been
around
for
a
long
time.
That's
one
of
the
examples
of
a
sketch
that
you
might
use
it's
computing
variable
with
histograms.
In
other
words,
all
right
last
slide
here
is
talk
about
work
in
progress.
C
We
aren't
quite
finished
here.
There's
there's.
Definitely
some
work
left
to
do.
One
of
the
things
that
we
haven't
tackled
yet
is
namespaces
most
of
your
metric
systems
out
there
let
you
have
namespaces
on
the
metrics
metric
instruments
and
we
see
that
as
being
a
requirement,
but
we
haven't
figured
out
exactly
what
the
API
should
look
like.
There's
some
moving
parts
in
the
open,
telemetry
specification,
there's
something
called
named
tracers
and
named
meters
is
maybe
related
or
maybe
not
so
we're
kind
of
still
working
on
that
there's
something
called
an
observer
gauge.
C
It
is
the
semantics.
Are
the
same
as
a
gauge,
but
it's
something
where
you
file
it
you,
you
register
a
callback
rather
than
calling
set
explicitly
and
there's
some
questions
about
what
the
calling
conventions
should
look
like
there,
but
these
are
actually
really
important,
because
if
it's
the
case
that
you
have
an
expensive
computation,
you
don't
really
want
to
have
a
timer,
that's
firing
any
more
frequently
than
metrics
collection.
C
So
if
you
have
an
expensive
gauge
and
there's
no
context,
meaning
you're
not
going
to
use
the
trace
or
the
distributed
correlations,
you
should
give
a
callback
and
then
the
metric
system
will
only
ask
current
value
when
it
needs
it.
That
way,
you
say
it
cost
so
that
we
see
this
as
being
really
important,
but
we
haven't
pinned
it
down
yet
and
then
the
current
SDK
that
we've
specked
out
or
the
example
that
we've
the
prototype
that
we
haven't
go
really
implements
several
fairly
simple
policies.
C
There's
one
batcher
that
does
aggregation
by
the
default
keys,
the
recommended
keys
that
we've
talked
about
and
there's
one
batcher
that
expore
it's
dated
by
all
the
keys.
Those
are
the
two
kind
of
like
extremes
that
you
might
have
in
a
simple
in
a
simple
system,
but
but
we
expect
that
that
real
systems
are
going
to
end
up
being
much
more
configurable.
So
you
can
imagine
getting
a
configuration
from
a
server
that
says
which
metrics
do
you
want?
Which
label
dimensions?
Do
you
want?
Which
aggregators
do
you
want
and
so
on?
B
C
That's
definitely
gonna
have
to
happen
and
I
think
that
probably
the
open,
telemetry
protocol
will
end
up
being
extended
to
provide
configuration.
It
happens
in
tracing
as
well,
where
you
say,
I
need
to
know
what
my
sampling
rate
should
be
because
I'm
because
I
don't
know,
and
so
that's
a
configuration
that
you'd
want
to
get
from
a
server.
So
very
much
likewise
in
metrics,
you'd
want
to
say:
hey
I
want
to
push
metrics
I,
don't
want
you
to
pull
them.
C
I
want
to
know
my
period
I
want
to
know
which
metrics
do
you
want
I
want
to
know
which
labels
you
want
so
on
so
I
expect
that
that's
coming
and,
lastly,
we're
still
working
on
the
standard
exporters.
We've
got
in
the
go
repository,
which
is
pretty
far
ahead
of
many
of
the
others.
We
are
working
on
Prometheus
and
staffs.
C
T
I
know
that
there
is
working
some
of
the
other
libraries
other
languages,
so
python
has
done
some
work
on
metrics
javascript
has
already
got
a
Prometheus
exporter,
and
so
on
so
we're
making
a
lot
of
progress
and
that's
where
we
are
and
that's
the
end
of
my
presentation.
Thank
you
all
very
much.
Thank.
B
C
Question
I've
been
talking
to
at
least
a
few
folks
from
the
open
metrics
project,
trying
to
figure
out
where
these
line
up
so
open
metrics
will
describe
itself
as
in
exposition
format,
meaning
it's
really
meant,
as
data
transfer
format
between
your
exporter
and
your
system,
so
open
metrics
does
have
a
summary
type
in
a
histogram
type,
so
we're
still
gonna
have
to
configure
what
export
do
you
want?
Do
you
want
the
histogram
output,
or
do
you
want
the
summary
output,
mm-hmm
and
so
open
metrics
will
make
a
great
data
transfer
format.
C
I
think
one
day,
I'm
waiting
to
see
the
RFC
that
they're
working
on
eagerly
actually
I
was
thinking
that
they
were
gonna
have
this
by
cube
con.
So
maybe
it's
out
there,
but
we're
waiting
to
see
that
open
telemetry
is
also
working
on
its
own
metrics
transfer
format,
which
is
part
of
this
ot
LP
protocol,
it's
structurally
very
similar,
so
we're
still
working
this
this
out.
C
I
think
that
the
I
think
that
these
two
protocols
are
going
to
be
very
similar
I,
like
the
open
metrics
format,
because
this
text
base
and
actually
text
can
be
a
really
fast
encoding,
much
faster
than
protocol
buffers
in
some
cases,
so
I
think
it'll
make
a
good
protocol.
It
needs
to
be
specified
and
I'm
waiting
to
see
the
the
current
draft
yeah.