►
From YouTube: 2022-06-16 meeting
Description
OpenTelemetry Prometheus WG
B
A
All
right,
I
also
reposted
the
I
renamed
the
initial
meeting
notes
from
the
first
meeting
into
just
a
general
overall
meeting
notes,
so
that
we
can
just
have
one
running
document
with
all
of
them.
Yeah
feel
free
to
add
yourself
to
the
attendees
list
and
then
also
there's
the
agenda
in
there
of
what
we
you
know.
I
guess
plan
to
talk
about
if
you
want
to
add
something
there
feel
free
to
add
something
as
well.
D
Can
you
link
it
in
the
meeting
request
in
the
meeting
request?
I
don't
know
if
I'm
allowed
to
change
that,
but
I'll
try?
Okay,
oh
okay!
We
also
posted
the
link
here.
Okay,
yeah.
I
was
just
looking
for
a
link.
That's
good.
A
I'll,
have,
I
think,
morgan
has
to
add
it
here,
but
I'll
have
him
at
it.
Yeah.
F
F
Awesome
got
that
all
right.
I
can
do
that,
but
I'll
do
it
in
the
background.
While
we
get
started
okay,
you
want
me
to
kick
it.
A
Off
yeah
that'd,
be
great.
All
right
looks
like
we
have
another
good
turnout
so
far.
A
Welcome
everybody
to,
I
guess
the
second
official
hotel
profiling
meeting,
basically
for
anyone
who
wasn't
here
at
the
first
one
we'll
do,
there's
a
recap
in
the
meeting
notes
of
sort
of
what
we
talked
about
last
time,
but
I
guess
to
give
a
quick
kind
of
overview
there's
been
there
was,
you
know
an
issue,
that's
been
that
was
created
in
the
hotel
repo,
a
while
back
about
adding
profiling
as
an
official
signal
type
for
hotel
and
there's
been
a
lot
of
conversation
there
and
in
the
recent
you
know,
I
would
say
a
year
or
two
years,
a
lot
of
progress
has
been
made
and
there's
been
a
lot
more
interest
in
profiling
as
a
signal
type,
and
so
after
some
discussion
at
cubecon
in
in
valencia
and
then
various
other
discussions
elsewhere.
A
There
was
enough
of
a
an
interest
in
kind
of
kicking
this
effort
off
and
officially
getting
the
ball
rolling
on
getting
profiling
as
an
official
event
type
for
hotel.
So
basically,
the
first
meeting
we
kind
of
just
got
collected
a
lot
of
people's
thoughts
on
various
ways
that
they're
using
profiling
and
how
they
would
like
to
see
it.
You
know
moving
forward
this
meeting.
A
I
guess:
there's
yeah,
there's
a
number
of
different
steps
in
order
to
get
it
officially
sort
of
accepted
as
a
signal
type,
and
this
meeting
is
to
kind
of
take
the
next
step
in
that
in
hopefully
coming
to
enough
of
a
consensus
to
start
creating
the
document
that
we
will
eventually
present
to
the
hotel
specifications
committee
and
and
other
groups
there
and
and
so
yeah.
So
that's
what
this
meeting
is
for
any
questions
on
that
before
we
get
started
or
thoughts,
okay,
so
yeah.
A
So
in
reviewing
the
last
week's
you
know
notes
there
were
some,
I
would
say,
like
the
main
points
that
people
talked
about,
were
you
know?
One
obviously
like
I
guess
the
ideas
to
consider
in
this
as
we're
trying
to
come
up
with
a
spec
is
one
you
know
being
able
to
conform
to
a
format.
So,
how
easy
is
that
format
for
existing
profiles,
profilers
to
kind
of
conform
to
there
was
talk
about
sending
events
in
a
non-file
format,
potentially
what
aggregation
level?
A
We
should
support
what
I
guess
yeah
what
should
be
included
in
this
schema,
what
fields
that
kind
of
stuff,
and
so
I
guess
to
kick
off
today.
I
know
there
was
also
some
conversation
since
then
in
the
hotel
profile,
slack
channel,
which,
if
you're
not
a
part
of
all
at
a
link
in
a
second,
but
I
believe
we
have
yeah
jonathan's
here.
He
was
part
of
that
conversation.
A
Not
to
put
you
on
the
spot
jonathan,
but
if
you're
willing
to
otherwise
I
can
summarize.
Basically,
there
was
some
discussion
around
stateful
versus
stateless
protocols,
which
is
relevant
for
symbols.
Encoding
has
a
lot
of
infra
implications
on
ram
network
bandwidth
stuff,
like
that,
I
don't
know
if
you
want
to
chime
in
with
your
thoughts
or
alexei.
Also
was
part
of
that
conversation
as
well.
A
If
either
of
you
want
to
either
sort
of,
like
recap-
or
I
guess,
bring
up
those
thoughts
here
and
see
if
anybody
else
has
any
comments
on
those.
D
Yeah,
for
this
is
alexi
for
profile
labels
yeah.
There
was
a
discussion
that
currently
people
for.
I
think
it
was
a
discussion
like
whether
people
format
would
be
suitable
and
what
extensions
people
kind
of
anticipate
for
their
needs
to
the
format,
and
one
discussion
was
about
profile
level
labels.
Currently,
there
is
only
stack
trace
labels
in
the
in
the
pro
format
and
in
google
at
google.
D
We
actually
discussed
having
profile
level
labels
several
times,
so
I
would
be
open
to
discuss
that
that
that
change
yeah
for
stateful
versus
stateless,
which
is
another
bullet
here-
maybe
sean
and
jonathan,
can
can
discuss
that,
because
I
can
only.
D
Give
my
interpretat
my
interpretation
of
of
of
that.
I
think
it
was
about
about
caching
of
like
on
one
hand
like
caching
of
the
symbolization
information
and,
on
the
other
hand,
minimizing
the
traffic
over
the
wire.
C
Yeah,
so
we
were
kind
of
getting
into
the
weeds
a
bit
around,
say:
yeah,
minimizing
traffic
over
the
wire
and
impact
on
ram
consumption,
on
both
the
the
back
end
and
also
on
the
agent
that
sort
of
thing.
C
But
when
we
were
talking
about
it,
I
think
the
kind
of
the
important
conclusion
for
me
that
came
out
of
that
is
that
it
wasn't
apparent
yet-
and
we
have
this
question
actually
in
the
document
like
what
problem
are
we
actually
planning
to
solve,
because
profiling
means
a
different
thing
to
a
lot
of
different
people
right,
you
have
everything
from
like
profiling,
just
one
application
on
one
machine
to
you
know
an
entire
kubernetes
cluster,
an
entire
data
center
and
depending
on
how
we,
which
subset
of
these
problems,
we
want
to
target
our
wire
format.
C
Our
overall
design
would
be
vastly
different.
So
if
you're
doing,
obviously
you
know
entire
data
center,
you
have
to
optimize
for
a
lot
of
different
things
you
wouldn't
have
to
optimize
for,
if
you're
just
doing
say
one
application,
a
good
example
might
be
say
whether
you're
sending
symbols
with
every
trace
or
not,
if
you're
just
doing
one
application
like
for
five
minutes,
it
doesn't
really
matter
if
you're
doing
5000,
nodes
and
they've
got
like
32
cores
and
they're
all
running
a
java
process,
which
has
you
know
very
simple,
heavy
data.
C
If
you're
sending
all
over
the
network,
it's
going
to
be
a
bad
time
and
so
yeah.
I
was
just
wondering
kind
of
like
what
what
thoughts
people
had
on
that
and
like
how
that
sort
of
discussion.
So
I've
never
been
involved
in
like
an
hotel
discussion
of
any
kind
before
I'm
not
really
sure
how
these
trade-offs
are
typically
are
made
or
how
the
decision
process
works.
C
So,
but
I
think
that's
kind
of
one
of
the
key
things
for
me
is
that
we
have
to
have
an
answer
for
that
before
we
start
really
talking
about
what
goes
into
the
actual
format
or
the
profiles.
A
Yeah,
so
I
guess
the
sort
of
the
choice
there
or
the
the
discussion
there
is
between
sort
of,
I
guess
yeah
like
instrumented
profiling,
or,
I
guess
more
of
a
sort
of
statistical
profiling.
I
I
don't
know
you
guys
think
that's
a
fair
way
to
do.
You
all
think
that's
a
fair
way
to
describe
the
two
sort
of
paths
that
we
could
go
down
there.
C
So
I
think,
even
in
that
instance,
we're
already
kind
of
into
the
description
of
how
we
might
actually
do
the
profiling
and
stuff
rather
than
if
we
talk
about
it
from
a
problem
point
of
view
like
what
problem
do
we
actually
want
to
solve
for
for
a
user?
Is
it
whole
system,
continuous
profiling,
entire
data
center,
or
is
it
individual
applications
say
for
a
small
amount
of
time.
G
Yeah,
I
think
that's
a
great
point.
I
know
here
from
pixi,
like
we
do
system-wide
profiling
and
some
of
the
points
that
you're
bringing
up
sean
are
very
relevant.
Like
we
care
about
the
format
I
mean
pete
on.
The
call
can
also
comment
on
about
like
symbols
like
there's
there's
a
the
volume
of
data
that
we'd
have
to
send
with
symbols
is
one
consideration.
The
other
is
also
if
you
don't
send
the
symbols,
whether
the
process
is
even
still
alive.
G
When
you
want
to
come
back
later
and
check
on
the
symbols
right,
so
you
have
to
store
the
symbols
somewhere.
There's
a
question
of
when
you
grab
the
symbols
where
you
store
them,
how
you
send
them
in
an
efficient
way?
I
think
those
if
we're
at
least
looking
at
system-wide
profiling,
I
think,
is
a
is
an
important
consideration
and
you
also
then
have.
H
Trade-Offs,
I
mean
sorry,
thomas,
go
ahead.
This
ties
in
with
a
question
of
status
about
the
state
full
because
I
guess
you
don't
want
to
rely
on
the
process
still
being
there
later,
particularly
if
you're
dealing
with
a
runtime
that
may
reach
its
stuff
in
the
meantime
and
so
forth.
C
And
I
guess
we
also
need
to
make
the
distinction,
then
between
say
for
native
code.
You
can
do
after
the
fact
symbolization
when
you
get
a
debug
package
right,
whereas
for
something
like
you
know,
java.net
python
php,
you
can
actually
pull
the
symbols
out
of
the
running
process
and
send
it
off
the
the
host.
So
I
don't
know
if
we,
when
you
think
about
these
things,
like
your
design,
gets
influenced
quite
heavily
and
yeah.
I'm
not
sure
if
we
have
any
answers
to
this
yet.
D
One
one
thing
I'm
curious
about
is
like
I
would
be:
we
probably
don't
have
it
anywhere,
but
I
would
be
curious
to
see
like
a
table
of
all
stakeholders
and
participant
and
interested
parties
in
this
discussion
and
what
format
they
are
currently
using.
I'm
curious
like
how
many
people
actually
use,
for
example,
p
pro
format,
how
many
people
use
gfr
format,
because
that
might
affect
the
dynamics
of
the
of
the
discussion.
I
guess
as
well.
A
Yeah,
that's
a
good
point,
maybe
I'll
just
put
in
the
what
format
are
you
using
I'll?
Just
put
it
in
the
meeting
notes
and
maybe
just
drop
in
there,
and
we
can
kind
of
get
a
informal
pull
there?
And
while
people
do
that,
pete
do
you
want
to?
You,
have
your
hand
raised
first.
I
I
I'm
curious
about
in
the
last
meeting
I
recall,
and
I'm
going
to
be
bad
about
the
name,
but
we
had
one
of
the
participants
mentioning
that
there's
already
a
generic
protocol
format
and
we
could
just
build
helper
methods
that
basically
encapsulate
the
profiling
data
into
that,
and
I
I
wanted
to
offer.
I
My
guess
is-
and
I
want
to
offer
my
own
hypothesis
is
that
there
is
some
something
about
sending
profiling
data
that
is
so
voluminous
with
symbols
and
the
amount
of
stack
traces
you're,
sending
that
it
might
be
nice
to
have
a
format
that
is
not
going
to
be
inefficient.
I
I
my
guess
is:
it
will
be
about
efficiency,
but
I'm
curious
to
see.
If
we
can
make
that
definitive.
I
I
I
thought
it
was
not
being
an
hotel
expert,
I'm
going
to
defer
to
that
comment,
but
hey
there
is
potentially
a
way
to
you
know
encapsulate
this.
The
other
comment
I
had
was:
I
wanted
to
offer
a
guess
that
the
interest
is-
and
this
is
informed
by
my
own
biases,
but
the
interest
may
lean
towards
you-
know:
scaled-up
statistical,
system-wide,
cluster-wide
data,
center-wide
style,
and
I
I
don't
fully
comprehend
the
nature
of
how
the
stateful
protocol
is
going
to
work.
I
I'm
curious
about
it.
It
it
sounds
like
it
has
some
vast.
You
know
amount
of
benefit
in
terms
of
reducing
the
data
transfer,
but
it
may
also
be
complicated,
so
I
don't
know
how
complicated
it
is,
but
those
are
kind
of
and
by
the
way
for
what
we
have
at
pixie
is
is
definitely
not
staple.
It
is
very,
very
simple,
in
fact,
so
we're
that
being
said,
I
so
my
bias
is
towards
actually
the
simple,
but
I'm
curious
to
know
about
the
the
like.
What
is
that
trade-off?
I
A
Does
anybody
want
to
respond
to
any
of
those?
I
guess
thoughts
on
on
that,
in
particular
before
we
go
to
felix,
do
you
want
to
respond
to
that
thomas.
H
Yeah,
I
think
I
agree
with
pete
that
both
like
the
efficiency
for
data
center.
A
So
real
quick
thomas
just
said
that
your
audio
is
pretty
low.
I
think
I
don't
know
if
it's
just
me
yeah,
it's
quite
quick.
H
Test
test:
better:
okay,
that's
better
yeah!
Yes,
I
agree
with
with
steve
that
the
efficiency
for
data
center-wide
profiling
is
really
going
to
be
a
kicker
or
important.
I'm
wondering
I
don't
know
much
about
pixi's
protocol.
I
do
know
a
bit
about
our
protocol.
Would
it
make
sense
to
have
both
pigs?
A
Yeah
I
mean
I
think
that
would
be
really
helpful.
I
mean,
I
think,
the
if
we
can,
you
know
hopefully
come
to
some
kind
of
consensus
here
on
on
what
goal
at
least
like
each
of
us
is
trying
to
solve
and
maybe
find
some
like
common
ground
there,
then
I
think
between
now
and
the
next
time
we
meet.
A
I
think,
a
good
you
know
actionable
next
step
is
sort
of
describing
either
the
different
yeah
yeah
describing
for
those
who
are
using
something
custom
like
why
you're
using
custom-
and
you
know
exactly
what
problems
the
existing
formats
did
not
solve
that
you're
solving,
and
I
think
that
would
be
a
good
sort
of
in-between
step
that
we
could
then
discuss
on
the
on
the
next
call
and
and
make
some
progress
there.
J
Okay,
okay
cool,
so
I
wanna
erase
a
point
that
was
also
mentioned
last
time,
which
is
signal
correlation
as
in
when
customers
are
already
instrumented
with
open,
telemetry
libraries
and
they
use
profiling
from
any
of
the
profile
inventors.
How
can
we
create
that
connection
between
the
profiling
data
and
the
open,
telemetry
tracing
data,
and
that
seems
to
be
a
neat
subset
of
the
problem
space
here,
where
we
could
maybe
agree
on
something
and
formats
that
works?
J
The
challenging
part,
of
course,
is
going
to
be
the
whole
host
profilers
that
are
that
are
not
deeply
embedded
in
the
runtimes.
How
can
they
get
trace
ids
van
ids
and
that
stuff,
but
that
might
be
a
smaller
problem?
That's
largely
also
organized
to
the
format
problem,
and
maybe
some
people
could
collaborate
on
this
one
specifically
as
well.
F
A
So
I
guess
I
mean
I
think
that's
I
would
do
I
guess
we'd
be
in
or
does
anybody
disagree
that
that
is
a
pro
I
mean.
I
think
that
that
it's
fair
to
say
that
that's
a
problem
that
would
that
this
format
should
solve
is
being
able
to
eventually
correlate.
You
know
profiles
with
something
else.
I
guess,
would
anybody
disagree
with
that?
Being
you
know
one
of
the
problems
that
we
hope
that
this
will
ultimately
solve.
A
Okay,
I
guess
we
we
got,
we
got
one,
we
got
one
for
the
for
the
problem
section,
I
guess
alexa,
you
got
your
hand
up,
go
for
it.
D
Yeah,
I
think
I
I
agree-
it's
a
good
problem
to
solve.
At
the
same
time,
I
think
there
are
details
behind
that
because,
for
example,
correlating
with
traces
does
it
mean
that
you
need
to
have
time
stamped
data
in
your
statistical
profile
and
that
kind
of,
like
immediately
increases
correlation
sorry,
it
increases
cardinality
of
of
the
data
so
like
how
statistical
the
data
is
and
how
time
exact
the
data
is.
That's,
that's
a
that's
just
a
trade-off.
It's
like
choose
details
or
compactness.
E
Hi
everyone
yeah,
so
a
couple
of
things
I
wanted
to
say
one
I
mean
hearing
from
everyone.
It
sounds
like
most
people
are
interested
in
data
center-wide
profiling.
E
So
I
almost
wonder
if
we
can
kind
of
decide
on
that
pretty
soon
as
well
as
opposed
to
doing
you
know
one
application
at
a
time.
That's
one
thing
and
the
other
thing
I
think
on
the
stateful
versus
stateless,
I
think
that's
a
pretty
big
and
how
we
handle
symbols,
and
things
like
that.
E
I
think
that's
a
pretty
big
kind
of
design
decision
that
we
need
to
make,
and
so
I
I
think
most
of
us-
or
I
think,
like
the
the
stateless
design,
is
kind
of
easier
to
understand,
and
I
I
think
most
people
have
some
intuition
on
how
that
would
work,
and
I
actually
think
speaking
of
presentations
and
things
like
that.
Maybe
it
would
be
more
interesting
to
hear
from
people
from
elastic
about
how
they
do
state
full
protocol
and
how
they
handle
symbols.
C
E
I
mean
when
I
say
that
I
mean
stateful,
meaning
you
you
know
in
in
repo.
In
one
request,
you
don't
send
the
whole
data
over
and
maybe
you
have
to
like
combine
it
with
symbols
after
the
fact,
and
so
you
have
to
keep
some
state
somewhere,
probably
on
both
client
and
server.
E
To
do
that
and
stateless
would
be.
You
know
you
send
the
whole
thing
every
time,
and
so
you
have
all
the
information
in
one
request.
That's
kind
of
how
I
look
at
it.
C
Makes
sense
yeah
I
mean
I
think
it
would
probably
be
it'll
become
apparent
like
what
we
do
isn't
that
complicated,
but
I
think
yeah,
it's
really
worth
us
like
doing
a
like
writing
a
doc
or
doing
a
presentation
or
something
and
hopefully
that'll
answer
some
questions.
A
A
Okay?
I
just
want
to
go
back.
I
added
it
to
the
doc,
but
just
want
to
make
sure
so.
Dimitri
mentioned
that
it
seems
like
here.
Most
of
the
interest
is
around
ability
for
data
center-wide
profiling.
Is
anybody
here
right
or
it
seems
like
the
primary
focus?
Is
that
would
anybody
you
know
any
thoughts,
the
opposite
direction,
or
anybody
disagree
with
that.
F
A
I
mean
yeah,
I
guess
it
seems
like
here.
What
people
are
saying
is
like
data
center
wide
being
you
know
some
sort
of,
I
guess
larger
scale
profiling
across
you
know
various.
You
know,
clusters
various.
You
know
yeah
data
center
wide,
whereas
the
opposite
of
that
being
just
like
profiling,
a
specific
application
for
a
period
of
time.
You
know
just
sort
of
more
of
it.
A
Every
function
called
yeah
yeah,
exactly
yeah,
something
that's
like
I
would
say
I
you
know,
maybe
arguably
but
less
suitable
for
more
of
a
production
workload.
J
A
Hold
on
give
it
one,
second,
for
your
audio
to
catch
up.
Okay,
I
think
you're
good
now.
J
Okay,
let
me
turn
up
video.
Maybe
that
helps
okay.
Can
you
hear
me
now?
Yes,
so
I'll?
Try,
the
the
thought
there
was
morgan
mentioned
that
maybe
on
service
instrumented
profiling.
You
might
have
to
like
put
a
lot
of
instrumentation
in
place.
I
would
argue:
that's
usually
not
the
case.
J
It's
a
very
small
amount
of
instrumentation
that
goes
into
service
profiling
and
I
can
speak
for
data
docs
if
you're
very
interested
in
that,
because
usually
it
allows
you
to
go
much
deeper
in
in
profiling
than
the
data
center
white
stuff,
which
is
mostly
focused
on
cpu
time.
So
I
think
I
want
to
say
we're
not
we're
interested
in
data
center
white
profiling
as
well,
but
I
I
would
vote
for
also
just
considering
the
per
service
level
and
also
point
out
that
the
instrumentation
is
likely
that
you
don't
need
much.
H
I
think
the
the
distinction
between
the
data
center
wide
profiling
and
the
pro
service
profiling
isn't
so
much
the
amount
of
instrumentation
required,
but
largely
the
amount
of
data
that
we
need
to
consider,
meaning,
even
if
we
like,
let's
assume
for
a
second,
we
had
a
way
of
doing
memory.
Profiling,
data
center
right
for
every
service.
H
The
distinction
isn't
so
much
for
how
we
obtain
the
data,
but
rather
the
quantity
of
the
data
that
has
to
be
considered.
If
that
makes
any
sense.
J
J
A
Oh,
I
think
we
might
have
lost
him
in
the
chat
thomas
and
dimitri
have
mentioned.
Maybe
this
making
the
distinction
between
continuous
versus
ad
hoc
profiling,
I'm
assuming
in
that
case,
meaning
you
know
profiling
consistently
versus
profiling.
Just
for
like
a
couple
minutes
or
you
know,
like
grabbing
a
profile
yeah.
F
B
That
we
will
impose
a
constraint
on
the
designing
of
the
format
for
two
types
of
format,
so
a
doc
versus
system-wide
profiling,
or
maybe
we
can
consider
the
ad
dock
profiling
a
subset
of
the
system-wide
one
or
vice
versa.
So
do
we
want
to
incorporate
this
distinction
in
the
design
of
the
format
or
is
like
more
on
the
collector
side
or
on
the
protocol
side?
A
E
I
think
maybe
the
the
perspective
I
was
coming
from
is
like
if
we
are
able
to
focus
on
one
that's
kind
of
more
important
and
to
to
kind
of
the
the
vast
majority
of
people,
and
it
sounds
like
this
data
center-wide
slash,
you
know
continuous
profiling
of
many
many
services.
E
That
seems
to
be
the
consensus
and
I
was
just
kind
of
thinking
like.
Maybe
we
should
define
that.
That's
what
we're
focusing
on
and
you
know
a
lot
of
conversations
become
kind
of
maybe
more
streamlined
and
but
but
yeah,
but
I
guess
we
could
potentially
make
ad
hoc
stuff
like
a
subset
of
it
or
or
not.
D
I
think,
as
long
as
it's
a
subset,
we
should
consider
it.
At
the
same
time,
there
may
be
types
of
ad
hoc
profiling
that
are
not
subsets.
For
example,
imagine
something
like
collecting.
I
don't
know
like
a
scheduling,
trace,
something
that
is
more
closer
to
the
debugging
scenario
and
that
might
have
like
it.
Might
those
scenarios
typically
allow
higher
overhead
if
you're
focused
on
functional
debugging
rather
than
performance
debugging,
they
can
also
produce
lots
of
data,
so
the
requirements
might
be
just
on
the
other
another
extreme.
D
You
want
like
extra,
for
example,
if
you
consider
let's
say
like
intel,
pt
data
performance
tracing,
that's
super
compact
because
it
doesn't
even
encode
kind
of
like
each
like
branch
addresses
it
will
only
encode
a
bit
for
whether
branch
was
taken
or
not,
which
requires
lots
of
I
mean
trade-offs
are
different.
I
think
when
you
slide
into
functional
debugging
scenarios,
so,
but
for
ad
hoc
scenarios,
which
are
kind
of
like
statistical
profiling.
I
just
want
to
get
to
this
specific
node
in
the
data
center
and
I
want
to
grab
a
profile
right
now.
A
A
I
mean,
I
guess
like
at
some
point
we
have
to
you
know
I
I
would
say
optimize
for
one
and
then
do
the
best
we
can
to
support
as
many
as
we
can,
and
from
that
perspective
it
seems
to
me
like
yeah,
this
sort
of
consent,
continuous
system-wide
profiling-
is
the
vast
majority
here,
but
that
they're
that
it
doesn't
necessarily
mean
that
we
wouldn't
be
able
to
with
what
we
come
up
here.
A
E
Maybe
one
last
thing
I
I'd
say
on
that
is
it
kind
of
sounds
like
this
continuous
data
center-wide
profiling
is
also
more
in
line
with
open,
telemetry
kind
of
thesis
or
ethos.
More
generally,
and
you
know
I
would
love
to
kind.
F
F
There
are
some
uses
of
open
telemetry,
starting
up
in
like
client
applications,
and
that
probably
will
be
a
bigger
thing
longer
term,
but
even
those
you
could
have
statistical.
You
could
apply
statistical
methods
too,
because
you
might
have
millions
of
millions
of
clients.
So
I
yeah,
I
completely
agree
with
demetri.
A
Cool,
I'm
just
gonna
move
what
formats
the
formats
part
down
into
here.
A
So
so
yeah,
I
guess,
are
there
any
if,
while
we're
selling
the
goals
point,
are
there
any
other
so
being
able
to
connect
profiles
with
other
hotel
signals
ability
to
for
data
center
system-wide
profiling,
with
continuous
as
a
primary
goal
and
ad
hoc
as
a
secondary
goal?
I
guess:
are
there
other
goals
of
problem
that
we're
trying
to
solve
here
that
are
not
included
on
that
list?
A
I
guess
even
if
and
and
one
thing
I'll
also
say
so,
tigran
linked
to
the
the
logs.
A
Let's
see,
I
don't
know
where
I
should
put
this
I'll,
just
put
it
in
the
chat
to
the
logs
documentation
or
the.
I
guess.
What
are
these?
The
the
data
model
for
the
logs?
I
don't
know
I
mean
I
imagine
some
of
these
may
be
able
to
to
this
is
like
similar
to
the
document
that
we'll
ultimately
have
to
be
able
to
create
here.
A
You
know
some
things
they
mentioned
there
being
able
to
map
existing
log
formats
to
the
new
log
data
model
or
the
otel
log
data
model,
it
being
semantically
meaningful,
translating
log
data
from
an
arbitrary
format
to
this
data
model
efficiently,
representing
the
data
model
so
yeah.
I
just
wanted
to
put
that
in
there,
as
we
could
potentially
steal
some
of
those
for
this
or
adapt
some
of
those
to
profiling
as
well.
Just
something
I
wanted
to
mention
there
sean
you
have
your
hand
race,.
C
Yeah,
just
one
other
point
that
maybe
we
could
go
one
way
or
another
one,
and
this
may
be
obvious
to
everybody
just
to
make
sure
we're
on
the
same
page
is,
I
guess,
when
we're
talking
about
doing
continuous
profiling
or
whole
system
profiling,
we
also,
I
guess,
want
to
account
for
say
just
the
ability
to
represent
traces,
that
aren't
just
say
through
one
runtime
or
say
through
just
native
code.
C
So,
for
example,
you
can
have
a
trace
that
might
start
in
the
kernel
go
up
into
userland
into
c
code
from
there
into
like
a
python
interpreter
into
actual
python
code,
then
into
some
other
c
code
and
like
you
can
have
yeah
like
stack
traces
that
are
going.
You
know
that
start
in
java
end
in
the
kernel,
but
in
between
there's,
like
you
know,
a
python
interpreter
and
a
whole
bunch
of
other
random
stuff.
So
we
may
well
we
not
that
we
may.
C
C
A
concrete
example
of
this
that
we
see
is
you'll
have
somebody
you
know,
writing
a
python
application,
but
the
actual
bottleneck
that
they'll
find
is
in
some
random
c
extension,
or
you
know
a
c
extension
that
calls
the
kernel
or
something
like
that,
and
so
you
want
to
be
able
to
represent
stack
traces
across
different
runtimes
in
native
code.
A
D
I
don't
know
if
it's
a
if
it's
a
requirement,
but
I
I
just
wanted
to
mention
that
is.
I
wonder
like
how
much
interoperability
with
the
and
like
existing
tools
outside
of
open
telemetry
world
is
important
and
what
I
mean
is
like.
If,
hypothetically
let's
say
like
we
end
up
with
a
format
that
is
like
like
80
like
gfr,
but
it's
not
gfr,
then
you
lose
the
interoperability
with
the
ecosystem
that
exists
for
gfr.
D
A
D
A
I
guess
I
would
be
interested
to
hear
I
mean.
Obviously
this
has
happened
with
metrics,
it's
hap,
you
know
like
there
was
existing
metrics
formats
and
yes,
hotel
came
up
with
one
I
seems
like
most
have
been
able
to
adapt
to
it.
So
I
guess
maybe
morgan.
You
can
provide
some
some
thoughts
there
yeah
so.
F
But
there
were
existing
wire
formats
right,
zipkin
had
one
jaeger
had
one
and
then
many
third-party
vendors
had
others.
Similarly,
for
metrics
otel
has
its
otl
people
wire
format,
but
it's
also.
We
also
offer
compatibility
with
prometheus
and-
and
so
it's
it's
probably
worth
discussing
this-
because
because
you're
right,
ryan
and
alexi
there's
there's
some
there's
the
same
sort
of
situation
here.
What
I
would
suggest
is
we
should
land
on
a
format
that
we
think
is
right
for
this.
F
For
example,
we
defined
a
metrics
data
model
that
was
compatible
with
prometheus
and
with
solutions
from
third-party
vendors
which
which,
by
the
way
like,
if
you
don't
know
a
lot
about
metrics
prometheus's
metrics
format,
is
quite
different
than
the
one
you
use
it
like,
google
or
datadog
or
splunk
or
various
others,
but
we
define
we
created
one
that
was
compatible
with
both
such
that.
Then
we
could
export
in
either
format
and
because
open
telemetry
has
exporters
that
can
be
attached
to
sdks
and
language
agents
and
to
the
collector.
F
I
would
suggest
that
we
define
a
form
format
that
at
least
can
be
relatively
trivially
turned
into
those
other
formats,
because
then
we
can
just
if
people
want
to
retrieve
something
in
jfr
or
prof,
then
great.
We
just
expose
it
as
that,
and
we
perform
like
a
pretty
trivial
conversion,
while
exposing
it.
F
F
A
Yeah
I
just
copied
over
what
they
had
in
the
logs
one
saying
you
know
it
should
be
able
to
map
to
existing
oops
profiling,
copy
pasta,
profiling
formats,
this
data
model
translating
back
and
forth
yeah.
I
mean,
I
think,
especially
as
you
know
popular
as
both
of
those
are
that
it
makes
sense
that
we
should
be
able
to
find
something
that
connects
them,
but
I
mean
I
think
I
don't
think
that
the
profiling
formats
that
exist
are
so
wildly
different
that
that's
you
know
a
hard
issue
to
solve.
Yeah.
A
F
With
tracing
that
was
really
simple
right,
like
like
the
the
data
model,
the
fundamental
underlying
data
model
used
by
like
zipkin
and
jaeger,
and
open
to
facing
open
census,
they're
all
like
all
sort
of
copies
of
the
stuff
published
by
google
like
10
years
prior,
and
so
that
was
pretty
straightforward
for
metrics.
That
was
actually
quite
challenging
because
prometheus
defines
metrics
in
a
very
different
way
than
most
other
other
people
or
organizations
that
consider
the
pre-debated
prometheus.
I
Yeah,
so
sorry,
I
missed
a
little
bit
of
the
meeting,
but
I
just
wanted
to
try
to
encapsulate
a
little
bit
of
what
I've
heard
or
what
I've
learned
in
this
meeting
is
we
want
to
sort
of
enumerate
existing
formats
and
taxonomize
them.
I
think
a
little
bit
so
I
I
just
want
to
say,
as
the
I
think,
there's
a
lot
of
interest
in
seeing
a
document
where
we
just
say:
here's
what
pprof
has
what
jfr
has
what
this
or
that
like
more
recent
vendor,
might
have
as
a
format-
and
I
just
want
to.
I
I
remember
the
discussion
about
what
was
stateful
versus
stateless
and
I'd
like
to
at
least
my
thought
process
is
about
having
the
taxonomy
like
well
understood,
so
that
so
that
there's
a
that,
for
my
purpose
would
give
me
like
the
mental
model
and
framework
of
thought,
as
we,
you
know,
talk
about
it
in
the
next
few
meetings.
A
Yeah,
I
think
that
makes
a
lot
of
sense.
I
guess
we'll
save
a
couple
minutes
at
the
end
here
to
come
up
with.
I
guess
yeah
like
what
the
to
do's
are
between
now
and
next
week,
but
of
current
profiler
profiling,
performance
and
so
yeah.
The
two
that
we've
talked
about
so
far
is
one
just
the
you
know,
yeah
like
understanding
the
current
formats
and
what
they
have
or
don't
have
and
then
two
for
custom
formats.
A
Those
willing
to
share.
You
know
more
explanation
about
what
is
custom
or
why
custom
over
existing
formats.
So
I
think
yeah.
I
don't
know
if
I
might
have
missed
something,
but
those
seem
to
be
like
the
two
biggest
things
so
one
just
like
understanding
the
current
profiling
formats.
A
I
guess
like
what
they
have
and
maybe
what
they
have,
what
they
lack
and
then
for
the
those
who
have
custom
formats
or
have
developed
custom
formats
in
their
organizations
being
able
to
share
as
much
as
you
can
with
those
of
what
is
custom
about
them
and
how
we
might
be
able
to
incorporate
that
into
a
you
know:
yeah
sort
of
collectively
agreed
upon
format,
dimitri.
E
E
I
think
it's
fine
if
we
convert
at
kind
of
like
render
time
when
we
show
you
know
data
to
to
people
when
you
know
they're
able
to
export
it
and
things
like
that,
but
converting
when
you're
generating
the
data
at
the
kind
of
the
client
level
application
level.
I
think
that
comes
with
potentially
a
performance
hit,
and
I
wonder
how
others
think
about
it,
because
to
me
that
seems
like
a
again
like
a
big
elephant
in
the
room
kind
of
a
big
deal
that
we
should
discuss,
but
maybe
I'm
missing
something
yeah.
A
I
guess
yeah
if
anybody
wants
either
sean
or
lexy
wants
to
respond
to
that,
and
I
don't
know
if
you
had
something
separate
alexa.
I
had
something
else
so
I'll
pause.
D
C
Yeah
sean,
yes,
I
agree
with
dimitri
in
the
sense
of
I
had
assumed
that
we
were
talking
about
conversion
on
read
conversion
on
right.
I
guess
yeah.
We
need
to
discuss
the
details
of
that
because
I
guess
yeah,
like
our
perspective
on
it
like
it's
always
been.
If
you
want
to
profile
everything
all
the
time
in
a
data
center,
you
want
to
be
doing
as
little
work
as
possible
on
the
on
the
host
and
depending
on
the
formats
like
it's
entirely
possible.
C
J
A
Yeah,
I
I
don't
suppose
many
would
want
to
do
conversion
elsewhere.
I
guess
if
anybody
does
feel
opposed
to
that
speak
now,
but
I
think
yeah.
I
think
it's
pretty
fair
to
assume
that
it
should
not
happen
on
the
client
side.
D
But
correct
me
if
I'm
wrong,
but
I
thought
open.
Telemetry
scope
is
mostly
kind
of
like
the
wire
itself,
for
example,
like
logging
format
or
tracing
format,
defines
how
how
the
data
is
represented
on
the
wire
so
like.
If
it's
converted
on
the
on
not
on
the
client
side,
does
it
mean
that,
like
then
like
how
the
date
is
represented
on
the
wire?
Does
it
means
like
it's,
not
open,
telemetry
wire?
I
I
don't
I'm
not
sure
who
I'm
I'm.
Probably
I'm
just
missing
something,
but
I
don't
quite
follow
it.
F
Typically,
this
results
in
the
the
sdks
and
other
components,
having
a
like,
obviously
they're,
in
different
languages,
but
having
a
basically
common
sort
of
set
of
objects
that
people
interact
with
in
process
and
then
also
we
have
the
single
otlp
format
for
each
data
type.
So
I
would
expect
with
profiling,
as
a
result
like,
whenever
we're
sending
anything
over
the
wire
in
otlp,
whether
it's
from
something
that's
capturing
profiles
from
java
or
go
or
c,
plus
that
that's
the
exact
same.
F
You
know
modular,
like
language,
specific
semantics,
but
like
it's,
it's
the
exact
same
compatible,
wire
format
and-
and
similarly
I
would
expect
like,
if
I'm
using
a
c,
plus,
plus
sdk
or
or
like
a
go
sdk
that
somehow
interacts
with
profiles.
F
If
we
define
that,
if
we
determine
that
such
interactions
are
necessary,
that,
like
roughly
the
same
sort
of
semantics
or
objects
or
mechanisms,
are
exposed
to
people
in
all
languages,
profiles
are
a
bit
different
than
traces
and
metrics
where
people
can
actually
interact
with
like
traces
and
add
annotations
directly
through
sdks
or
profiles.
Perhaps
we
just
decide
yep
they're
captured.
You
can't
actually
interact
with
them
beyond
that,
though,.
D
It
made
it
made
sense
to
me,
I
think,
also.
For
example,
we
had
we
had
stateless
versus
stateful
discussion
and
in
that
discussion
I
also
kind
of
assume
that
it
means
that
they're,
like
four
months,
are
coming
like
we're
discussing
the
client
side
as
well,
because
otherwise,
like
the
whole
stateful
versus
stateless,
is
kind
of,
like
only
makes
sense.
If
you
talked
about
about
like
client
server
interaction,
why
would
we
discuss
stateful
versus
stateless?
If
we
only
talk
about
conversion
and
read
time.
D
And
kind
of
related
to
that
as
part
of
taxonomy,
we
discussed
like
taxonomy
for
formats
and
when
people
use
custom,
why
they
use
custom.
I
would
actually
love
to
also
learn
about
taxonomy
of
the
actual
continuous
data
center
profiling
in
terms
of
like
sampling
rates.
How
often
machines
are
profiled
things
like
that,
because,
because
continuous
like
it,
you
can
call
many
things.
D
Continuous
continuous
could
be
like
every
machine
at
the
data
center
100
time
in
cloud
profiler,
for
example,
we
profile
if
there
are
like,
for
example,
let's
say
like
there
are
1
000
replicas,
then
each
particular
replica
each
particular
server.
We
would
profile
fairly.
Rarely
that's
why
I
was
also
like
when
this
whole,
like
stateful
versus
stateless
discussion,
was
when
we
were
talking
about
that.
I
was
like,
like
I
wonder
why
it's
important,
because,
for
example,
in
the
product
that
I'm
working
on,
we
don't
profile
the
same
server.
D
Consequently,
that
often
so,
there
is
not
really
a
lot
of
state
like.
Are
we
talking
about
state
that
is
somehow
shared
between
replicas,
but
then
I'm
curious
how
it
works,
or
is
that
particular
profiler
profiles
like
almost
hundred
percent
of
time
everything?
But
then
I'm
curious
like
how
do
they
deal
with
with
overhead,
because
even
if,
like
continuous
profiling
could
be
like
one
percent
or
two
percent,
then
at
very
large
deployments.
That's
like
two
percent
fleet
wide
is
a
lot
of
money.
D
Like,
like
a
short
summary
of
of
like-
and
we
could
pick,
I
don't
know
like
an
abstract
example-
it
program
deployments
in
three
data
centers
across
three
different
continents,
two
thousand
replicas
each
like
what's
the
profiling
rate
for
different
profilers,
look
like
or
something
like,
I'm
not
like
saying.
This
is
a
good
example.
Just
just
having
something
concrete
and
having
people
respond
to
support
would
be
nice.
C
That's
a
good
question
like
I
do
think
some
of
this
will
become
clear
when
we
kind
of
like
do
some
sort
of
like
doc
or
presentation
on,
because
that
will
cover.
I
think
these
things,
because
they're
all
good
questions,
but
I
guess
we
might,
I
think,
we've
probably
all
made
different
design
decisions
on
the
answers
to
those
questions.
Yeah.
D
C
Just
one
question
in
terms
of
how
we
want
to
convey
this
information
like
in
the
group,
would
it
be
better
for
us
to
write
a
document
like
a
google
doc,
and
then
we
have
a
discussion
like
this
afterwards
or
put
together
like
a
presentation
and
like
discussion
live?
Does
anybody
care
I
mean
a
doc
is
obviously
allows
us
to
be
more
verbose
and
more
detailed.
I.
A
A
Yeah
explain
live
over
one
of
these
meetings.
I
think
that
would
be
useful
and
honestly
yeah
for
anyone
because
yeah
I
mean.
I
think
that
this
even
this
particular
question
is
kind
of
hard
to
put
in
a
poll.
It's
like
somewhat
of
a
yeah.
It's
a
a
somewhat
philosophical
thing
of
like
what
is
continuous.
Like
you
know,
oh,
was
that
technically
continuous.
Is
it
not,
and
so
yeah
I
mean
I
that
would
be
my
boat.
I'm
curious
if
other
people
have
other
thoughts
there.
F
Be
useful,
I
suspect
the
various
implementations
we're
discussing
are
actually
more
similar
than
than
we
might
think,
but
but
yeah
right
ryan.
I
you
know.
Actually
I
agree
like
maybe
just
like
everyone
has
like
a
short
paragraph
or
maybe
like
ryan.
If
we
can
come
up
with
like
five
five
or
six
questions
that
we
that
for
everyone
that
if
they
answer
it,
that
will
sort
of
allow
people
to
understand
where,
where
they're
coming
from.
A
A
Yeah,
so
we've
got
five
minutes
left
here.
So
I'm
wondering
in
terms
of
next
steps.
A
Yeah
so
that
definitely
sounds
like
one
so
yeah,
like
one
being
the
text.
Taxonomy
of
you
know
current
profiling
formats.
You
know
what
they
have,
what
they
lack
and
then
for
custom
formats
for
those
willing
to
share.
So
how
about
it
sounded
like
both
thomas
and
sean
were
willing
to
share
some
information
about
sort
of
what's
custom
and
I
believe
both
of
you
mentioned
statefulness
or
I
guess
like
thoughts
around
statefulness.
Would
you
be
willing
to
to
write
something
up
related
to
that
and
share
at
the
next
meeting
yep.
A
C
Work
so
the
other
thing
is,
I
just
need
to
make
sure
that
everybody's
comfortable
with
sharing
all
of
these
details,
like
I
I
assume
they
are,
but
let
me
just
run
that
by
some
people,
that's
fair.
A
Sure-
and
I
don't
think
you
necessarily
need
to
tell
us
like
yeah-
I
don't
know
to
share
about
custom,
how
about
just
share
thoughts
on
custom
format,
difference
between
custom
and
current
formats.
I
guess.
A
And
then
same
with
thomas,
I
guess-
or
I
guess
thomas
maybe
had
to
drop,
you
know
yeah,
if
you're
not
able
to
that's
or
was
it
was
it
thomas
or
was
it
maybe
pixie
who
had
mentioned
it?
A
I
guess
let
me
look
who
who
said
they
had
the
custom
formats.
I
guess
I'll
just
put
it
this
way,
anyone
who's
willing
to
share
about
the
custom
formats.
Who
wants
that
to
be
a
part
of
this
conversation,
I
guess
we'd
kind
of
have
to
know
sort
of
just
like
from
a
high
level
sort
of
like
in
what
way
you
know.
You've
made
your
formats
custom
so
that
we
can
kind
of
apply
that
to
this
I
guess
that's
kind
of
the
the
ultimate
goal
there,
brian.
I
I
I
had
mentioned
we,
we
can
add
the
details
of
our
format.
It
is
custom,
it
is
not
going
to
be
the
most
advanced
one
out
there.
The
I
just
wanted
to
tie
up
the.
I
I'm
glad
to
hear
like
this
talk
around
taxonomy.
The
reason
there's
a
specific
reason
I
have
which
is,
after
we
all
add
paragraphs
that
are
kind
of
verbose
about
what
what
it
is
the
if
the
taxonomy
is
is
good.
Then
you
can
put
it
all
in
at
the
table.
I
That's
compact
and
sort
of
that's
sort
of
like
my
in
in
my
back
pocket.
I'm
like,
I
hope,
to
see
it
all
encapsulated
in
one
table.
It
might
not
totally
get
there
because
the
taxonomy
may
not
cover
all
the
bases,
but
once
once
it's
established,
that's
so
incredibly
useful
for
at
least
from
where
I
stand.
A
Fair
enough
so
yeah,
so
maybe
what
we
can
do
is
like
in
the
in
the
slack
channel.
We
can
kind
of-
I
guess,
take
this
thing
about
it
a
little
bit
and
come
up
with
maybe
like
a
couple
of
sort
of
things
to
evaluate
the
various
profile
formats
on
you
know
like
ability
to
add
you
know,
I
guess
related
to
the
goals,
I
guess
would
make
sense
ability
to
connect
to
other
signals
ability
for
data
center-wide.
You
know
representing
transmitting.
You
know
like
basically
evaluate
them
on
those
four
points.
A
I
guess
yeah
we'll
definitely
need
to
take
some
time
and
sort
of
like
clean
that
up
a
little
bit
or
make
those
as
distinct
from
each
other
as
possible
and
then
and
then
that'll
kind
of
give
us
something
good.
We
can
move
forward
with
there.
I
think.
F
That'll
be
good,
and
then
the
other
thing
I
would
say
just
scenario-wise
like
in
addition
to
the
formats,
is
just
talk
of
when
we
write
this
stuff
down
to
talk
a
bit
about
like
how
you're
collecting
these
profiles
or
how
you
expect
them
to
be
used.
Like
specifically,
what
I'm
trying
to
touch
on
here
is
like
there
used
to
be
a
lot
of
some
profilers.
That
would
just
be
like
instrumenting
every
function
call
and
capture.
Basically
traces
of
function
calls.
F
My
sense
is
that's
not
as
popular
most
people
are
now
doing
sampling
profiles,
profilers,
where
the
the
data
has
to
be
aggregated
or
inspected
over
time
to
be
useful,
just
because
of
the
perf
impact
of
of
instrumenting
every
function
call
that's
my
assumption,
but
I
want
to
make
sure
that's
clear,
so
we
should
also
note
that
down
for
each
because
if
the
first
is
out
there,
then
you
know
that's
that's
useful
information
for
this
process.
A
Okay,
so
yeah.
Obviously
this
was
I
mean.
I
think
this
was
a
good
discussion.
You
know
we
kind
of
moved
around
a
bunch
in
the
discussion,
so
I'll,
try
and
like
you
know,
clean
up
and
organize
the
notes
from
today
and
then
repost
in
the
slack
channel,
both
this
video
once
it's
posted
and
then
also
sort
of
what
the
to-do's
are
as
and
what
we
plan
to
yeah.
I
guess
how
we
can
assign
those
and
plan
to
talk
about
them
for
next
week.
A
How
does
that
sound
to
everyone?
That
sounds
great
all
right
cool?
Well,
thanks
for
coming
again
everybody
and
have
a
good
rest
of
your
week
slash
weekend.