►
From YouTube: 2023-03-21 meeting
Description
Instrumentation: Messaging
A
A
A
People
a
minute
or
two
more,
it
looks
like
Tristan,
went
ahead
and
added
at
least
some
topics,
but
ideally
he
said
what
a
join
and
actually
talk
to
them.
A
A
I
think
bossy
mentioned
there's
a
lot
from
the
like
AWS,
like
our
ADOT,
like
Lam
delay,
release
process,
so
we
can
potentially
reuse
and
Tyler
I
know
you've
looked
into
release
tooling
and
then
Mike
I
know
you've
had
some
concerns
about.
You
know
getting
changed
logs
versioning
and
things
like
that.
So
I'm
hopeful
to
just
kind
of
use
this
item
to
be
a
generic
like
catch-off.
Whatever
like
release
work,
we
need
to
be
like
that
needs
to
be
done.
A
So
I
guess
you
know
with
that
context,
yeah.
What
do
you
guys
got?
I
heard
that
I
missed
or
more
in
depth.
A
Well,
I
guess
we
need
to
think
through
exactly
what
we
need
in
place
before,
like
the
releases
happen.
So
like
the
change.
Log
in
version
is
like
one
example
of
that,
and
then
the
tooling
is
a
kind
of
like
a
separate
thing,
so
I'm
just
trying
to
make
sure
we
we've
captured
everything
we
need
to
implement
to
get
to
like
that
first
release
and
do
that
in
a
good
state.
D
C
D
Your
first
questions
we
probably
need
to
answer
is:
what
is
it
we
intend
to
release?
Presently
the
ADOT
Lambda
layers
are
combined
Auto,
instrumentation
layers
and
collector
extension
layers,
with
the
exception
of
the
one,
that's
just
the
collector
extension
layer
for
go
and.net.
But
we've
talked
about
wanting
to
separate
that
I
know.
There
are
some
users
who
are
exporting
to
otlp
destinations.
Don't
want
to
do
any
translation
in
The,
Collector
extension
and
for
them
an
auto
instrumentation
layer
would
be
perfectly
sufficient.
A
And
in
the
layers
today,
though,
you
can
turn
off
the
the
instrumentation
correct
or,
if
just
like,
baked
into
it,
I'm
assuming
there's
some
sort
of
like
lever
to
flip.
D
It's
pretty
well
baked
in.
It
has
to
do
with
how
you
set
the
the
Handler,
whether
there's
a
wrapper
script
that
you
invoke,
or
about
to
start
up
the
the
application
or
not.
E
I
kind
of
think
it
makes
sense
to
have
the
collector
be
a
separate
layer
or
extension
from
the
the
the
individual
language
releases.
E
I
know
that
that
creates
kind
of
a
hard
dependency
requirement,
but
if
somebody,
for
example,
configures
their
exporter
within
their
app
to
use
something,
that's
non-oldtp,
you
know
in
theory
they
could
send
it
directly
to
an
external
search
system.
E
But
by
separating
it
it
allows
us
to
kind
of
version
them
independently
release
them
independently
and
not
couple
them
so
tightly.
D
Yeah
I
think
the
the
only
concern
I'm
aware
of
that
could
arise
from
keeping
them
separate
is
that
there
is
a
limitation
on
the
number
of
layers
that
can
be
attached
to
a
function
instance,
which
I
believe
is
five,
so
auto
transportation
and
collector
would
consume
two
of
those
I'm,
not
sure
how
much
of
an
issue
that
is
in
practice.
D
E
F
D
E
D
Think
it
would
have
like
a
significant
cost
in
terms
of
storage
or
service,
or
anything
like
that.
The
the
cost
is
going
to
be
cognitive,
you
know,
does
it
cause
confusion
to
users?
Does
it
drive
support
requests
for
which
of
these
do
I
use?
How
do
I
make
work
sort
of
things.
A
E
D
My
inclination
and
what
I've
been
advocating
internally
for
the
eight
hours
as
well,
is
to
keep
them
separate
so
that
users
can
choose
Auto,
instrumentation,
choose
collector
or
choose
both.
E
D
Something
I
could
point
it
at.
You
know
a
vendor
endpoint
or
an
internal
service
that
they're
running
or
wherever
it
needs
to
go.
E
And
the
main
cost
with
that
is
it
taking
two
layer
slots
instead
of
one
and
do
you
think
that's
the
a
significant
issue
for
people.
D
I
I
haven't
seen
any
situation
where
it
would
be
an
issue.
In
my
experience
in
providing
support
for
land
users
I've
not
seen
more
than
two
layers
in
use
anywhere.
Okay,.
E
Yeah,
in
my
experience,
it
wasn't
too
that
that
limitation
wasn't
as
annoying
as
the
only
two
log
filters
limitation.
E
For
for
the
log
forwarding,
so
I
I'm,
okay
having
it
be
separate.
D
Yeah
I,
my
hope,
would
be
that
the
clear
documentation
can
address
that
because,
like
the
customer
is
going
to
have
to
make
a
change
to
choose
to
go
from
the
AI
layer
to
the
hotel
layer
in
the
first
place,
also
as
I
say,
I'm
advocating
internally,
that
we
make
this
change
for
the
8
out
layers
as
well.
So,
if
I
think,
if
open
Telemetry
makes
that
choice,
it
becomes
easier
to
make
the
case
that
it
ADOT
should
also
be
doing.
That.
E
The
other
consideration
is
when
you
have
things
like
platform
dependencies
so,
for
example,
x86
or
or
arm
64
kind
of
choices
that
in
the
the
layers
must
be
compiled
specifically
for
that
version
or
sorry
for
that
platform,
if
it's
relevant.
So
if
you've
got
something
like
the
auto
instrumentation
that
likely
isn't
necessarily
going
to
be
as
platform
dependent,
I
believe
that
the
collector
is
correct.
D
Correct
yeah
the
collector's
always
going
to
be
platform
dependent
because
it's
compiled
in
native
code,
I
think
for
auto
instrumentation
or
SDK
layers.
Python
is
the
only
one
I'm
aware
of
that
has
any
platform
dependence
that
comes
from
it's
grbc
export,
which
depends
on
compiled.
E
Code
so
I
guess
the
main
thing
there
is.
That
increases
the
the
number
of
distributions
that
would
need
to
be
published
by
coupling
them
together,
as
opposed
to
keeping
them
independent.
D
A
D
A
A
A
D
Versioning
I
think
would
be
worth
a
discussion
I
think
right
now,
the
8
out
layers
are
kind
of
inconsistently
named
We,
inherited
from
the
the
group
that
created
them.
Initially
a
scheme
where
the
auto
instrumentation
layers
are
named
after
the
version
of
the
API
or
SDK
that
is
used
in
that
layer
of
the
collector
generic
collector
layers
named
after
reversions,
with
the
the
version
number
of
The
Collector
that
it
contains
there's
no
kind
of
concept
of
versions
of
the
Lambda
layers,
independent
of
what
they
contain.
D
So
if
we
do
separate,
then
like
how?
How
does
someone
know
if
a
Lambda
layer
for
auto
instrumentation
is
compatible
with
the
layer
for
The,
Collector,
Etc
and
there's
there's
more
to
what's
included
in
those
Auto
implementation
layers
than
just
the
API
or
SDK,
which
are
sometimes
at
very
different
versions?.
A
That's
a
good
issue:
wouldn't
the
SDK
or
The
Collector
to
speed
a
dependency
of
like
the
certain
layers.
You
would
have
the
dependency
version
supported
as
part
of
that
release,
or
that
version.
D
Yeah,
that's
that's
my
thinking
is
that,
as
we
start
these,
you
know
Hotel
cncf
released
layers.
We
start
from
zero
zero
one.
You
know
start
a
brand
new
versioning
sequence
and
their
their
versions
based
on
the
layer
version,
and
then
we
keep
in
the
changelog
and
release
notes
and
all
of
those
places,
information
about
what
SDK
API
instrumentation
versions
they
contain
Etc.
A
A
D
E
Because,
like
in
the
Java
repo,
the
Java
agent
itself,
the
auto
instrumentation
itself
has
a
single
version,
which
obviously
is
composed
of
other
things
within
it
that
have
independent
versioning.
D
Right
and
that's
that's
the
place
where
relying
on
the
embedded
instrumentation
version
might
make
sense,
but
because
it
doesn't
make
sense
for
Python
and
JavaScript
and
I'm,
not
even
not
sure
about.net.
What's
going
to
happen
with
it
when
they
add
Auto,
instrumentation
or
Ruby
as
far
as
I
understand,
I
would
I
would
prefer
consistency
across
the
rather
than
trying
to
do
what
works
for
one,
and
you
know
independently
in
isolation
for.
E
Lambda
layers,
one
thing
I've
seen,
is
that
it
tends
to
have
less
semantic
versioning
and
follows
more
of
a
just.
A
sequential
release.
Number
kind
of
method
is.
C
D
It
just
for
you
automatically,
you
upload
a
new
layer
version
and
it
increments
that
number
I
think
I
would
be
fine
with
that
as
well.
E
D
Okay,
in
that
case,
we
would
start
from
one
when
you
upload
the
first
one.
It's.
A
Okay,
so
from
a
release,
toin
perspective,
so
Anthony.
What?
What
do
you
all?
Having
like
your
internal,
a
DOT
like
subscriptions
and
things
like
that
set
up
around
released
a
processes?
I'm
assuming
you
have
like
a
test
and
Dev
or
a
test
and
production
environment,
and
then
I,
don't
know
some
sort
of
integration.
Testing
too.
D
Yeah,
so
basically,
the
way
it
works
is
there's
integration
test
account
where
the
the
CI
process
builds
layers
puts
them
into
the
integration
test,
accounts
the
in
A,
non-publicly,
Accessible
Manner
and
then
runs
a
set
of
tests
against
that
with
functions
that
are
also
constructed
in
that
account,
and
then
we
we
put
a
hash
of
the
or
the
the
commit
hash
in
the
layer
name
so
that
we
can
easily
identify
different
layer
versions
based
on
you
know
what
commit
was
was
last
tested
and
then
at
release
time
we
basically
say
take
the
layer
that
has
this
commit
hash
in
it
and
go
move
that
into
the
release
location
with
whatever
version
number
it
ended
up
becoming.
D
So
CD
is
basically
just
pick
it
up
from
here
and
move
it
over
there.
After
all,
the
testing
is
completed.
A
A
Tyler,
do
you
ever
get
a
chance
to
do
some
of
like
the
kind
of
POC
work
in
your
own
subscription
I?
Remember
you
may
be
doing
something
back
in
February,
but
not
sure
how
far
you.
A
D
I
think
it
would
largely
be
a
matter
of
taking
the
GitHub
actions
workflows
that
are
defined
in
the
repo
I'll
link
them
over
evaluating
what
needs
to
change.
I
think
there's
some
assumptions
there
about
having
two
separate
accounts
or
releasing
in
tests.
D
We
presently
only
have
the
one
account
I
think
yeah,
so
we
might
need
to
do
something
in
terms
of
permissions
on
test
layers
to
ensure
that
they're
not
generally
accessible
until
we
decide
to
make
them
publicly
accessible
during
the
release
step,
but
I
I
would
think
largely
those
those
kind
of
actions
can
be
lifted
and
shifted.
A
Yeah
I'm
not
super
hopeful
around
getting
a
test
account
just
because
getting
the
one
account
with
like
pulling
teeth
from
multiple
people,
so
I
get
I,
guess
I'll,
try,
but
we'll
see
what
the
the
response
is
like.
Okay,
thanks
thanks
for
clarifying
that.
A
A
D
I
think
things
get
a
lot
clearer
if
we
separate
The
Collector
from
the
auto
instrumentation,
because
then
the
collector
layer,
we
release
every
time,
there's
a
collective
release.
We
just
follow
that
up
and
then
the
instrumentation
we
can
release
either
on
a
periodic
basis,
if
the
relevant
sake
also
releases
on
a
periodic
basis
or
whenever
we've
decided
that
there's
enough
change
accumulated
in
in
those
eggs
to
work.
E
Would
it
make
sense
to
automate
that
such
that?
If,
for
example,
when
the
collector
does
their
release
to
automatically
push
a
new
version
of
the
layer.
D
I,
don't
know
that
I
can
be
fully
automated,
at
least
not
at
this
time,
because
the
collector
still
makes
breaking
changes
that
need
to
be
incorporated
into
what's
included
in
the
layer,
but
it
certainly
can
be
added
to
the
collector's
release
process
where
one
of
the
last
steps
of
the
collector's
release
process
is
to
notify,
for
instance,
the
the
operator
that
that
they
need
to
make
a
new
release.
D
We
do
the
same
for
Lambda
layer
and
as
long
as
the
built-in
release
process
is
mostly
automated,
it
should
be
a
matter
of
making
a
PR
to
update
the
collector
dependencies
and
deal
with
any
breaking
changes
and
then
kick
off
the
test
and
release
process.
B
A
Okay,
so
what
do?
What
does
the
word
breakdown
look
like,
potentially
for
the
converting
the
GitHub
actions
and
then
starting
like
the
instrumentation
separations,
I
know,
Python
and
JavaScript
are
always
probably
going
to
be
P1
I'm
here
and
then
we
can
probably
Focus
on
Java
after
that
and
separating
that
I'm
curious.
If
you
want
to
start
releasing
some
of
these
layers.
First,
like
a
JavaScript
one
or
the
python
one
on
its
own.
A
We're
gonna
wait
until
you
know
that,
before
or
like
the
current
poor
layers
are
all
ready
to
be
released
at
once
kind
of
curious
group
thought.
C
E
I
I
agree:
I,
think
that
should
be
a
prerequisite,
because
you
can't
really
do
the
other
languages
until
you
have.
The
The
Collector
is
like
an
independent
layer.
A
B
A
Okay,
that
makes
sense
we
can
probably
get
some
people
from
the
lights
up
side
to
do
some
of
this,
and
if
any,
would
it
make
more
sense,
for
maybe
some
of
the
AWS
folks
to
cover
the
GitHub
action
conversion
or
what?
What
is
y'all's
availability
contribution
lines
here.
D
I
think
personally,
my
availability
is
not
going
to
be
very
high,
but
I
will
touch
base
with
the
team
and
see
the
folks
who
are
working
internally
on
our
release
process.
What
their
availability
is
for
contributing
Upstream
as
well.
A
Okay,
yeah,
that
would
be
great
I
know.
Bossy
has
mentioned
some
stuff
before
so
maybe
just
yeah
I
guess
Circle
back
with
him
and
see
his
interest,
but
great
okay
and
Tyler
I
know
you
have
a
project
board.
Do
you
wanna
or
do
you
want
us
to
add
some
of
this
stuff
to
it
or
I
want
to
glance
at
the
one
time?
I'll
be
honest.
E
This
I
created
that
one
just
to
kind
of
help
keep
all
of
the
the
various
different
issues
and
PR's
collected
together
onto
one
view.
So
it's
easier
to
see
them
all
in
one
place.
A
Okay,
I'll
go
ahead
and
I
guess
create
issues
for
all
of
like
these
action
items
at
least,
and
we
can
start
tracking
them
in
the
actual
land
area,
though,
and
that'll
that'll
at
least
be
a
start.
C
C
That
you're
gonna,
because
in
the
open,
Telemetry
Lambda
repo
each
of
these
languages
and
The
Collector,
are
just
a
subdirectory
in
there
is
the
idea
that
you
would
have
a
change
log
in
each
of
those,
because
it's
because
the
way
Adar
does
is
it
release
once
a
month
and
it
just
so.
Here's
our
our
March
release
and
here's
all
the
different
versions
of
it.
But
if
we're
going
to
release
things
on
different
cadences,
that
wouldn't
make
sense.
D
A
D
Indeed,
is
a
very
strong
word:
I,
don't
think
it's
needed
here.
It
may
make
some
things
easier,
but
it
would
primarily
add
a
bunch
of
overhead
for
other
things
as
well.
A
Yeah
I
don't
see
some
of
the
like
Community
governance,
people
being
super
enthused
about
like
a
new
repos,
so
just
from
that
perspective,
I
think
keeping
it
to
one
would
be
preferable.
A
A
C
Some
of
them
so
I'm
not
maybe
there's
a
way
to
do
this,
but
is
there
a
way
to
get
notified
when
a
change
log
is
updated
like
so
because
if
you
have
your
own
repo
and
a
new
releases
published,
you
can
get
a
notification
on
that.
So
clients
can
be
aware:
hey
there's
a
new
version.
Can
you
do
that
same
thing
in
any
capacity
with
a
change
log
file
never
tried
before.
C
Because
the
main
thing
that
I,
like
me,
working
with
customers,
I,
want
to
have
the
awareness
for
the
customers
to
know
that
hey
something
new
is
released.
This
may
fix
some
problem
that
you've
been
having
in
the
past
and
then
the
other
side
is
like
in
the
change
logs.
The
problem
I
have
with
the
change
logs
today
is
it
says
it
contains
like
open,
Telemetry,
python
version
1.16,
but
that's
not
a
link.
So
it's
like
how
do
I
know
what
open
Telemetry
python
116.?
C
C
And
this
came
about
because
I,
just
I
updated
my
layers
to
the
latest
version
and
all
of
a
sudden
I
started
getting
it
broke
like
everything
started
complaining
about
metrics
instead
of
the
other
and
I
had
no
indication
that
there
was
going
to
be
any
type
of
breaking
change
in
there
and
I
haven't
investigated
that
further,
so
I
don't
know,
but
but
but
that's
kind
of
like
where
I'm
getting
it
is.
E
So
I
would
say
that
the
the
change
logs-
maybe
you
can't
really
be
alerted
on
that,
but
I'm
pretty
sure
you
should
be
able
to
get
similar
kind
of
alert
from
setting
up
a
GitHub
release
notification.
E
C
D
Well,
I,
don't
think
that's
a
decision
that's
been
made
yet
I.
Think
that's
one
of
the
the
questions
is:
what's
our
release
Cadence
right
and
if
that's
a
thing
that
factors
into
it,
if
having
disjoint
releases,
would
be
problematic
for
communicating
change
to
users,
then
that
may
advocate
for
having
a
unified
release
Cadence
for
all
of
the
large.
A
Yeah
we
could
potentially
just
do
it.
I
know,
ADOT
has
their
own
kind
of
process,
and
you
know
concerns,
but
I
mean
we're
bumping
it
down
to
bi-weekly
instead
of
monthly
I
mean
I,
know
like
frequency,
has
been
a
concern
for
some
of
these
updates,
but
that
might
be
one
way
to
do
it.
The
community
Smooths
at
a
faster
Cadence
on
the
releases.
A
C
D
B
E
Use
releases.
A
A
A
Okay,
yeah
fair
enough,
so
I
guess
we
still
potentially
I
I,
guess
Mike!
If
we
have
those
GitHub
releases,
though
it
does
having
the
unified
change.
Log
still
be
kind
of
a
requirement
for
you
or,
if
you're
able
to
get
GitHub
release
notifications
and
we
can
do
it
on
a
less
consistent
basis
or
more
link
dependent
basis.
C
A
C
Yeah,
that's
part
of
my
biggest
thing
right
now
is
just
having
to
look
everything
up
and
go
through
it
to
identify
I
realize
you
can't,
could
pull
all
that
information
in
yourself
to
say:
here's
all
the
changes,
but
if
we
can
make
it
so
it's
clickable
so
that
you
can
get
to
them.
That
would
be.
That
would
be
good.
A
A
Alex
or
Tyler
are
we
kind
of
curious?
What
are
your
thoughts?
Are
Anthony
I,
don't
know
if
they're
gonna
be
some
sort
of
GitHub
tool
or
something
like
that,
but
it'd
be
nice
to
automate
it
to
some
degree.
Otherwise,
it's
just
kind
of
a
maintainer
or
approver
like
standard.
You
know
enforcement
type
of
thing.
E
I
support,
cross-linking
and
and
backlinking
for
that
I
think
if
we
get
it
set
up
with,
you
know
the
release
templates,
then
that
should
make
the
process
a
little
easier.
B
A
So
I
do
see
the
value
of
it
Mike.
It
just
seems
like
right
for
automation
or
extremely
painful
for
repo.
B
C
C
C
Yeah
not
not
do
it
on
a
different
path,
but
a
lot
of
what
we
find
with
customers
is
there's,
there's
a
bug
or
something
that's
been
addressed
in
the
API
somewhere,
and
that
creates
a
pull
request.
And
then
you
you
look
at
that.
That
whole
request
has
been
merged
to
follow
that
path
from
where
that
pull
request
has
been
merged
to
when
it's
been
released
in
the
Eight
dollar
error
or
the
open
Telemetry
Lambda
is
difficult.
It's
really
hard
to
figure
out
when
that's
actually
been
been
released
and
that's
what
they're
waiting
on.
C
E
One
of
the
things
that
I
really
appreciated
at
datadog
is
we
used
milestones
on
our
issues,
so
if,
for
example,
a
customer
reported
an
issue
or
a
particular
PR
got
merged
when
it
was
released,
a
milestone
representing
that
release
was
added
to
that
particular
PR,
so
that
it
was
much
easier
and
more
visible
where
that
PR,
which
which
version
that
PR
was
merged,
was
released
into
and
that's
something
I've
kind
of
missed
having
in
the
at
least
in
the
the
Java
repo
I.
E
A
E
Ahead,
I
was
gonna,
just
try
to
follow
up
a
little
bit
on
one
of
Tristan's
questions,
one
of
the
things
that
we've
been
a
little
bit
confused
on
this
is
specifically
the
environment
variable.
So
going
back
to
the
environment,
variable
question
that
we've
decided
to
shift
over
to
a
span
link
I,
don't
fully
understand
the
life
cycle
of
that
environment
variable
Anthony,
maybe
you
can
help
me
understand
a
little
bit.
How
does
that
get
set?
And
what
is
the
purpose
of
it
is
it?
E
Does
it
represent
a
span
for
the
the
entire
instance,
or
is
it
per
request
and
it
somehow
gets
updated
on
each
invocation.
D
My
understanding
is
that
it
is
per
request
and
is
updated
on
each
invocation
by
the
runtime,
but
I
don't
have
visibility
into
the
labor
runtime
to
confirm
that
right.
E
Now,
okay,
do
you
know
what
it's
for?
What
is
the
purpose
of
that.
D
E
Okay,
so
when
it
does
should
the
should
they
be
identical.
D
E
E
That
invocation
should
have
a
parent
that
would
so
the
trace
ID
should
be
identical,
but
the
the
span
ID
will
be
different
and
within
the
X-ray
system
it
should
eventually
point
back
to
the
original
SDK
invocation
correct.
D
If
x-ray
Trace
complex
propagation
is
used
in
the
event
right
and
and
if
that
Trace
context
is
provided
to
the
exec
API,
when
the
event
when
the
invocation
happens,
because
there
are
right,
there
are
two
places
where
that
context
needs
to
to
go.
One
is
in
the
event,
if
you're
using
the
event
to
carry
the
context,
and
the
other
is
to
the
Lambda
API.
So
the
Lambda
API
knows
how
to
construct
the
spans
or
the
segments
that
it's
going
to
create
and
produce
to
x-ray
I.
E
Right
by
event,
I
meant,
like
the
the
system
property,
the
the
system,
where
the
basically
the
the
AWS
x-ray
context,
as
provided
via
HTTP
header
to
the
outgoing
AWS
request,
SDK
request.
E
I
I
think
that
helps
me
understand
a
little
bit
better.
The
main
issue
with
that
particular
span
is
that
it
is
only
available
to
X-ray
and
not
available
to
other
systems,
correct.
E
E
E
At
least
from
from
that
context,
with
what
Tristan
was
asking
so.
A
Cool?
What
do
you
call
it
here?
Thanks
guys
for
joining
I,
get
the
issues
created
and
we'll
start
scheduling
some
of
the
work
on
our
side?
Anthony.
If
you
can
Circle
back
with
your
team
and
just
figure
out,
if
there
is
any
availability
to
help
on
the
release
tool
and
that
would
be
that'd,
be
cool.