►
From YouTube: Cloud Native Live: Open Telemetry
Description
Don't miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe in Amsterdam, The Netherlands from 18 - 21 April, 2023. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
A
A
They
will
build
things,
they
will
break
things
and
they
will
answer
all
of
your
questions,
so
you
can
join
us
every
Wednesday
to
watch
live
so
this
week
we
have
Jessica
here
with
us
to
talk
about
open,
Telemetry
and
as
always,
this
is
an
official
live
stream
of
the
cncf
and
as
such
it
is
subject
to
the
cncf
code
of
conduct.
So
please
do
not
add
anything
to
the
chat
or
questions
that
would
be
in
violation
of
that
code
of
conduct.
B
Thanks,
yes,
hi
I'm,
jessatron
I
am
currently
in
developer
relations
at
honeycomb,
but
I'm
a
software
developer
of
20
years
and
development
is
what
I
care
about,
and
today
today
I
have
a
program
that
I
want
to
change.
It's
it's
just
this
script
that
I
wrote
like
at
Christmas
time
last
year
for
this
cute
little
Christmas
activity
and
I
want
to
make
it
more
General.
B
B
I
voted
for
fun,
I,
didn't
write
tests,
and
so
it's
really
hard
to
refactor
things
without
tests,
because
you
never
know
when
you
break
something
and
yeah,
you
could
like
make
a
small
change
and
then
run
it
again
and
see
if
the
output's
okay,
but
that's
so
tedious,
also
it's
hard
to
write
tests
for
because
it's
like
a
visual
thing.
It
invokes
Randomness.
B
It
uses
stuff
relative
to
the
current
time,
so
I'd
have
to
like
pull
all
those
things
out
of
the
functions
that
are
like
reading
the
file
system
and
reading
the
clock
and
and
calling
Rand
I'd
have
to
pull
all
that
out
to
make
it
have
deterministic
tests
at
all,
and
that
would
be
a
big
refactoring,
but
I
can't
refactor,
because
I
don't
have
tests.
B
B
Well,
such
as
production
is
with
a
script
that
you
run
on
your
computer,
but
in
real,
but
in
in
other
code.
If
you
practice
this,
those
traces
will
still
be
useful
in
production.
They'll
be
useful.
Every
time,
I
work
on
this
program
in
the
future
and
the
other
reason
is.
It
actually
makes
something
change
with
my
refactoring.
B
So
my
goal
today
is
to
add
some
tracing
and
see
something
that
I
want
to
refactor
and
then
refactor
it
and
see
the
results
in
the
traces,
even
though
it's
not
externally
visible
the
program's
just
still
working
the
traces
are
supposed
to
change
as
the
internals
of
our
program
change,
sometimes
as
we
choose,
and
so
that's
going
to
be,
the
feedback
loop,
okay,
Annie
can
I
share
my
screen.
What
do
I
need
to
do?
Yes,
perfect?
Oh
there,
it
is
yeah,
okay,
okay
right!
So
here's
my
little
main.ts.
B
This
program
already
has
open
Telemetry
configured
for
reasons
that
will
become
apparent,
but
it
doesn't
have
tracing
around
its
functionality
yet
so
when
in
doubt
or
always,
do
the
thing
and
then
see
what
it
see,
what
it
doesn't
do
and
then
make
a
change,
so
I
can
run
it
in
local
mode,
where
it's
sending
Telemetry
to
a
local
collector
and
a
Jaeger
instance,
and
then
I
can
go
over
to
Jaeger.
And
what
I'm
looking
for
here
UI
find
traces
is,
as
is
a
span
with
a
top
level
Main.
B
It
doesn't
have
a
main
there's
only
this
one
Trace
and
if
I
click
on
this
you're
going
to
see
what
this
program
actually
does.
So
what
this
program
actually
does
is
output
a
trace
with
pictures
in
it.
Let
me
shrink
this
a
little
and
you
can
see.
Can
you
tell
that's
what
that's
supposed
to
be?
B
It's
actually
supposed
to
be
a
Christmas
ornament,
and
this
this
is
supposed
to
be
a
Christmas
tree
and
there's
another
ornament,
another
ornament
or
whatever
kind
of
ball-like
thing
you
want
that
to
be
smaller
tree
ornament,
bigger
right,
so
it
draws
pictures
in
the
traces
it
does
some
other
stuff
and
at
the
very
end,
hopefully
I'll
have
time
to
pop
over
to
Honeycomb
and
show
you
like
the
entertaining
things
that
this
program
does,
that
with
with
open
Telemetry
spans
that
aren't
visible
in
Jaeger.
B
But
for
now
it's
enough
to
see
in
here
that
it's
still
working,
it's
still
making
trees
and
bobbles
inside
the
trace.
I
see
this
little
line.
That's
a
span
event!
That's
supposed
to
be
a
sparkle
right,
but
we're
not
getting
traces
from
the
program
describing
its
own
execution
and
that's
what
I
want
to
start
doing
before
I
refactor.
A
Yeah
and
we're
getting
a
few
comments
from
the
audience
that
the
screen
is
looking
a
bit
blurred
just
to
let
you
know
the
audience
is
what
we
try
to
troubleshoot
it
and
make
it
less
blurred.
We
could
not
get
it
fully
working
as
we
would
have
liked
to,
but
it
was.
It
is
the
best
that
we
could
do
at
the
moment.
A
So
I
hope
it's
working
for
everyone,
but
if
there's
something
that
that
you
would
like
for
us
to
zoom
in,
for
example,
more
or
something
let
us
know
but
yeah,
let's
hope
they.
B
Works,
you
can
also
do
this,
for
you
Branch
name,
add
tracinglive.
So
if
you
want,
let's
see
if
you
go
to
I'll,
just
open
it
on
here,
you
want
the
code.
It's
available
at
honey,
github.com,
honeycombio,
slash,
happy
Ollie
days
where
Ollie
is
011y
and
it's
the
the
ad
tracing
live
Branch.
That's
where
I'm
working
from
right
now,
if
you
want
to
go,
look
at
it
yourself
and
someday
I'll
have
to
figure
out
how
to
get
restreamed
to
show
my
screen
differently,
but
that
could
be
that
could
be
ours.
A
Yeah
we
worked
really
well,
the
difference
is,
is
better
than
the
first
time.
Okay,
cool.
B
B
That's
its
game,
because
it's
a
script
and
not
like
a
long-running
program
like
a
web
application
I
have
to
explicitly
start
the
SDK
and
wait
for
that
to
complete
before
I
do
any
operation
that
could
create
a
trace.
Otherwise,
you've
got
a
trace,
ID
full
of
zeros,
and
it
you
don't
get
a
Trace
and
after
that,
I
have
to
explicitly
call
SDK
shutdown
before
the
program
exits
or
it
just
won't,
send
all
the
traces
at
the
end.
B
So
that's
something
to
know
about
tracing
in
a
batch
job
or
script,
something
that's
not
a
web
app.
That
runs
forever
so
now
to
add
a
trace
at
all
I'm
going
to
wrap
this
main
call
in
a
trace
and,
let's
see,
do
I
have
a
tracer.
Yet
I
don't
have
a
tracer.
So
let's
get
one.
B
Okay,
const
Tracer
equals
hotel
that
tray
stuck
at
Tracer,
we'll
name
it
main.js,
because
that's
the
file
we're
in
now
I
can
say:
tracer.start
active
span,
you
have
to
say
start
I'm
a
JavaScript
I
mean
node,
you
have
to
say
start
active
span,
not
just
start
span,
that's
important,
because
if
you
want
your
child,
your
span
that
you
create
to
ever
have
children.
B
It
has
to
become
the
active
span
and
the
problem
with
that
is
that,
in
order
to
have
an
active
span,
you
have
to
put
all
the
code
the
unit
of
work.
That's
going
to
be
inside
that
Span
in
a
callback,
bummer
callbacks
and
the
Callback
receives
the
span
as
an
argument.
B
So
we
can
call
from
there
we
can
get
main
main
is
an
asynchronous
function.
It's
returning
a
promise.
So
after
main
completes,
then
I
can
say
s
dot
end.
You
have
to
end
the
span
yourself
and
today
you're
going
to
see
the
travails
of
adding
tracing
parts
of
it
are
easy
parts
of
it
are
hard.
Mostly
if
you,
if
you
know
how
it's
working
then
you're
fine,
that's
kind
of
asking
a
lot
too.
B
B
So
now,
instead
of
just
calling
rain
I'm
calling
Maine
inside
an
active
span
and
I
end
that
span,
it
doesn't
get
sent
until
you
call
end.
So
it's
really
important
to
call
end
in
order
to
get
spam
all
right.
Let's
run
this
unlocal
and
this
time
I
expect
to
go
to
Jaeger
and
see
a
whole
new
Trace
up
or
oops.
Okay,
I
passed
the
wrong
thing
to
record
exception.
I,
don't
know
why
it's
not
telling
me
that
in
here,
whatever
okay
record
exception,
once
the
error,
not
the
spam,
that
makes
sense.
B
Thank
you,
typescript.
Okay,
this
time
it
ran
the
same
good,
go
over
to
Jaeger
and
see
if
we
got
a
new
Trace
find
traces
all
right.
We
have
one
with
552
spans,
that's
going
to
be
like
the
one
that
it
created
with
pictures
in
it,
but
we
also
have
a
trace
with
a
single
span
and
its
name
is
Maine
and
that's
what
I
wanted?
B
Okay
there.
It
is
it's
it's
a
main
span.
It's
it's
not
very
exciting,
but
it's
a
starting
point.
Okay,
the
next
thing
to
do
is
what
Auto
instrumentation
can
I
add,
because
this
is
no
there's
got
to
be
some
automatic
instrumentation.
That
I
can
add
to
find
out
kind
of
spy
on
what
this
program
is
doing,
and
that's
going
to
tell
me
something
about
the
program
I'm
modifying
as
well
as
giving
me
a
starting
point
for
the
spans
that
I
can
use
to
describe
my
refactoring.
B
I
could
also
grep
or
Import
in
source
star.ts,
but
I
want
to
exclude
local
Imports
grab.
Dash
V.
B
So
I'm,
excluding
quote
Dot
okay
clear:
do
that
again,
no
did
the
grab
again
there
we
go
all
right
at
the
top
here.
I
know
it's
not
super
clear
I
can
make
it
even
bigger.
I,
don't
know
if
that
helps
at
the
top
here
it
says:
import
axios
from
axios
all
right.
So
axios
is
a
it's
a
HTTP
client
I'm.
Using
that
to
call
out
with
HTTP
calls
that's
something:
I'd
want
to
track.
B
Pngjs,
that's
obscure,
that's
not
going
to
have
Auto
instrumentation
and
the
file
system
that
looks
useful
and
then
I
have
open
Telemetry
and
then
I
have
file
system
again,
open,
Telemetry,
open,
Telemetry,
open
telemetry
and
the
word
important.
Okay,
so
I
want
Auto
instrumentation
for
axios
and
file
system
and
when
I
want
out
of
instrumentation
I,
always
go
to
open
Telemetry
dot,
IO,
slash
registry
registry
and
I
scroll
down
until
I
find
the
search
box
and
then
I
type
axios.
B
No
matches
found
done
what
do
we
have
for
node?
So
if
I
look
at
JavaScript
and
type
instrumentation
and
I,
look
at
the
okay,
I
can
get
the
whole
list
of
JavaScript
implementation
and
I.
Did
this
yesterday
and
I
looked
and
looked
for
something
that
would
be
relevant
because
I'm
like
well,
if
axios
is
an
instrument
that
I
could
use
a
different
library
that
is,
and
here
I
found,
HTTP,
instrumentation
and
HTTP
is
like
the
built-in
node
library
that
does
Network
calls
and
axios
is
going
to
use
that
under
the
hood.
B
And
then
great
I'm
going
to
add
it
to
my
tracing
configuration.
B
Here's
my
tracing
configuration
I
have
set
the
logging
to
tell
me
if
something's,
wrong
and
I
have
the
SDK
set
up
with
a
batch
processor,
where
I've
like
upped
the
limits
on
everything,
because
generally
the
the
the
libraries
try
not
to
interfere
with
the
functioning
of
your
program.
But
in
this
case
open
Telemetry
is
the
functioning
of
this
program.
So
I'm
like
please,
yes,
use
my
memory,
but
then
I
don't
have
any
instrumentations.
So,
let's
add
those
instrumentations.
B
B
B
So
when
I
click
on
that,
so
here's
my
main
span
and
here's
an
HTTP
get
and
another
one
and
another
one
so
I
can
see
that
we're
making
three
Network
calls
in
this
app
if
I
click
into
these
and
click
into
the
tags
and
dig
and
dig
and
dig
I
can
find
that
the
target
here
is
the
off
endpoint
at
honeycomb.
So
it's
this
program
calls
into
honeycomb
to
be
like
hey.
B
B
So
the
the
first
one
and
third
one
here
are
duplicates
and
I
could
definitely
shave,
270
milliseconds
off
the
execution
of
this
program
by
saving
the
output
of
the
first
get
call
here.
That's
one
thing
to
learn:
that's
cool:
let's,
try,
adding
a
file
system
instrumentation
so
for
the
file
system.
Instrumentation
I,
go
back
to
the
registry
and
I
look
for
the
file
system.
Fs
instrumentation
FS
is
built-in
node
library
for
file
system,
and
this
should
tell
me
how
to
use
it.
B
So
let's
go
through
that
game
again:
open
Telemetry,
instrumentation,
FS
installs
that
and
then
we
go
to
tracing.ts,
and
this
is
something
this
one
gets
me
a
lot
of
value
for
a
little
bit
of
typing
experimentation.
B
Edition
fs
and
then
we'll
add
it
to
the
array
here
and
we'll
have
two
Auto
instrumentations
and
that's
going
to
do
a
lot.
Two
auto
instrumentations
run
it
again
and
then
we'll
go,
look
at
Jager
and
see
what
files
this
this
is
opening.
B
Jaeger
back
find
traces
and
aha
there's
a
big
one
and
a
small
one.
I
want
the
small
one.
No
I
don't
want
the
small
one.
I
want
it
to
be
bigger
than
that
one
span,
four
spans
I,
don't
have
it
yet.
Maybe
if
I
hit
find
traces
again,
oh
wait.
Here's
one
of
51
spans.
Let's
try
that
one!
B
Oh
okay!
So
here's
our
main
and
look
at
all
this
at
this
real
path.
Sync,
whatever
that
means
stat
sync
read
file:
sync
blah
blah
blah
read
file.
Sync
read
file.
Sync:
read
file.
Sync
read
file,
sync,
okay!
So
we're
doing
a
whole
lot
of
synchronous
file
reads:
what
file
are
we
reading
if
I
click
on
one
of
those
bands
and
I
click
into
the
tags?
It
tells
me
nothing.
It
tells
me
nothing.
This
Auto
instrumentation
is
unsatisfactory.
B
B
So
that's
a
bummer
if
I
do
want
to
find
out
what
files
it's
reading
and
I.
Also
find
it
interesting
how
scattered
the
file
reads
are,
for
my
purposes
of
making
this
configurable
I'm,
probably
going
to
want
to
move
those
closer
to
the
beginning
of
the
operation
of
the
program.
B
Okay,
so
I
can't
really
modify
the
auto
instrumentation.
That
would
be
hard,
but
I
can
put
spans
around
the
read
file.
Syncs
that
I'm
doing
I
go
over
a
vs
code
and
I
look
for
read
file.
Sync
I
find
not
markdown
that
doesn't
matter.
Okay,
I
find
image.ts
and
song.ts.
A
Oh
yeah,
so
they
asked:
do
you
guys
have
any
representations
for
doing
the
same?
On
our
end,.
B
Of
it
yeah,
if
you
want
to
look
at
the
code
for
this
example,
it's
at
I
have
that
open
over
here,
nope,
that's
seven
telemetry,
that's
open,
telemetry,
honeycombio,
slash,
happy
holidays
is
currently
where
this
code
is
in
general
I'm,
using
a
zillion
open,
Telemetry
projects.
B
So
this
is
this
is
all
a
bunch
of
Open
Source
stuff
that
I'm
using
and
if
you
want
this
particular
example
of
using
it
that's
available
on
GitHub
too
and
Carlo
I'm.
Sorry,
I
can't
make
restream
work
better
for
the
screen.
Clarity,
someday,
I'll,
try,
but
that's
out
of
for
like
10
minutes,
right
I'm
on
a
Mac
Pro.
This
is
not
a
weird
computer
anyway,.
B
Do
you
want
to
like
make
progress
quickly
or
do
you
want
to
see
see
adding
tracing
be
hard
foreign.
A
Let's
see
what
people
say,
my
gut
feeling
would
be
the
hard
way.
That's
always
the
more
entertaining
way,
but
let's
see
does
anyone
else
agree
with
me
or
not.
A
A
Thank
you
a
lot
for
the
links
that
you
provided.
So
that's
helping
me.
B
Okay,
well,
if
nobody
has
an
opinion
or
maybe
there's
just
a
bit
of
delay,
then
I-
oh,
oh,
okay,
Rob
G,
says
the
hard
way.
All
right.
The
hard
way
is
we'll
start
with
the
hard
way.
So
here
in
song.ts
I'm,
like
maybe
maybe
it's
this
one
we
probably
are
gonna
need
to
instrument
both
of
them.
But,
okay,
here
we
go.
We
have
our
read
file
sync.
So
what
I
want
to
do
is
wrap
the
body
of
the
function
in
a
start
active
spam.
B
To
do
that.
I
need
a
tracer.
I
have
a
shortcut
for
that
get
Tracer,
so
we
will
get
Tracer
named
song.ts
because
that's
the
file
name
and
that's
my
personal
preference
I
need
to
get
this
file
name
onto
a
spam.
Okay,
so
normally
what
I
do
is
I
use
my
span
shortcut
and
I'm.
Like
return,
Tracer
dot
start
activespan.
It's
immediately
angry,
because
this
is
a
Constructor,
so
I
can't
return
anything!
Oh
okay!
Well,
that's
fine!
I'll,
just
wrap
oops
name!
This
band
Jess
span
song
I'll
just
wrap
here.
B
B
Well,
we
have
the
bane
of
every
JavaScript
program,
which
is
the
word
this,
and
this
is
not
in
fact
this
once
I
put
it
inside
this
callback,
foreign,
so
frustrating
I
fought
with
this
for
a
while
earlier
and
gave
up
and
did
the
easy
one
and
it
turned
out
to
be
useful
but
I
think
in
general.
What
what
I
need
to
do
here
is
actually
put
this
stuff
in
a
utility
function.
B
B
B
Instead
of
putting
that
file
read
directly
in
the
Constructor,
I
think
that's
going
to
work
better
because,
specifically,
once
it's
in
a
function,
I'm
no
longer
doing
anything
with
this
now
I
can
do
the
standard
method
of
wrapping
the
body
of
the
function.
In
start
active
span,
oops
I
forgot,
to
name
it
name.
It
read
file
for
song
all
right
and
then
I'll
scoot
the
end
of
the
function
down
here.
Oh
s,
dot
n
needs
to
be
right
before
the
return.
B
B
Okay,
now
I
expect
at
least
some
of
my
read
file
syncs,
not
all
of
them
to
have
another
span
on
top
of
them
and
that
span
should
reveal
to
me
the
file
name.
That's
happening
or
that's
being
read:
okay,
so
I've
run.
It
again
takes
a
couple
seconds
to
make
the
cow.
This
is
all
startup.
That
cow
is
super
early.
Okay,
it
did
some
stuff
back
to
Jaeger
back,
find
traces,
see.
B
I
had
51
spans
now,
I
have
52
spans,
that's
a
good
sign
when
I
click
into
this
Trace
I'm
going
to
scroll
down
and
look
for
what
did
I
call
that
something
about
a
song
there.
It
is
this
says,
I'm
sure
it's
super
tiny
because
Laurie
screen,
but
this
is
read
file
for
song
and
it
has
a
file
name
input,
slash,
song.txt,
okay
and
I
can
tell
that
it's
taking
place.
It's
not
quite
the
last
file.
That's
read,
but
it's
close,
there's
one
other
one
somewhere,
that's
kind
of
interesting.
B
That
surprises
me,
but
that's
interesting.
Okay,
so
there's
a
read
file
for
song,
but
it
only
showed
up
once
there's
only
one
occurrence
of
song
on
the
screen,
so
that
was
that
was
the
hard
one,
but
it
only
got
us
one.
So
let's
go
do
the
other
one
image.ts
there's
another
read
file.
Sync,
this
one
conveniently
is
in
a
function,
not
a
Constructor,
so
this
should
be
easier.
B
Make
a
tracer
image.ts
this
the
name
of
this
Tracer
is
going
to
show
up.
Let
me
show
you
where
that
shows
up,
because
I
find
it
interesting.
It's
here
under
tags
for
the
span.
We
have
otel
Library
name,
and
that
says
song.ts.
B
So
that's
where
the
Tracer
name
winds
up
in
this
case.
That
tells
me
that
the
because
I
named
it
this
that
tells
me
which
file
the
the
span
was
created
in
also
I
can
search
for
its
name
in
the
code.
B
All
right
we've
got
this
Tracer
and
now
we're
going
to
use
it
to
wrap
the
body
of
this
function
in
a
tracer.
So
we'll
call
this
one
read
PNG,
because
this
is
reading
a
PNG
file
and
then
move
this
to
the
bottom.
It's
really
important.
One
reason
I
put
this
in
the
Alias.
Is
you
have
to
return
the
output
of
start
active
span,
or
else
it
gets
munched
and
your
function
doesn't
return.
Anything
probably
just
typescript
gets
mad
at
you
if
you're
in
JavaScript,
that
can
be
really
tricky.
B
B
B
B
Thanks
David
for
the
bless
you
all
right,
Yeager.
What
do
you
know?
B
Fine
traces?
What's
at
the
top
86
fans,
86
bands
that
says
something
all
right.
So
here
we
are
at
the
top.
We've
still
got
these
read
file
syncs
the
beginning,
spoiler,
those
actually
wind
up
inside
the
first
HTTP
call
or
the
first
axios
color
I
should
say.
But
here,
aha
now
we
have
read
PNG,
we
have
a
bunch
of
read:
PNG
spans,
read:
G,
read
PNG.
B
The
browser
tells
me
that
I
have
34
of
those
34
VPN
G's
fans,
wow;
okay!
Well,
which
files
are
we
reading
app
a
file
name
input,
slash,
don't
Peak:
okay,
app.file,
name
input,
input,
slaus
input,
slash,
bigger,
tree.png,
bigger
tree
again,
bigger
tree
again,
bigger
tree
again:
okay,
it's
it's
clearly.
B
A
There's
a
link,
the
question
from
Carlo,
which
is
amazing.
Thank
you
so
much
for
great
questions,
everyone.
It
looks
like
you're
modifying
the
production
code
with
tracers.
What's
the
overhead,
you
are
adding
how
you
measured
that
how
to
evaluate
the
threshold?
Where
is
work
at
this
traceability
operation
they're
imagining
a
fast
and
quick
set
of
light
operations?
What
would
be
the
impact.
B
Okay,
so
adding
tracing
to
production
code,
there
are
people
who
measure
the
impact
of
this.
It's
very
small
tracing
Precinct
production
code
is
like
a
modern
standard
for
production
code,
because
in
this
case,
it's
just
giving
us
a
picture
of
what
one
program
is
doing,
but
in
a
distributed
system
it
gives
you
a
picture
of
the
overall
system
tracing
in
production
when
it's
carefully
curated
like
this,
is
you
get
like
a
strong
picture
of?
B
What's
important
in
your
app,
an
extreme
level
of
tracing
would
be
profiling
and
profiling
is
it's
known
to
add
overhead
and
production,
but
but
these
days
we
have
continuous
profiling
where
that's
pretty
low
overhead,
too
tracing
is
less
overhead
than
profiling
generally,
because
it's
it's
not
as
high
resolution
I'm
only
tracing
the
units
of
fork
that
I
said
are
important:
a
file
system
access.
B
Let's
see
how
long
that
takes
actually
how
much
this
file
system
access.
This
one
is
taking
273
273,
microseconds
wow,
that's
really
small,
actually,
that
the
whole
re
well.
Okay,
that's
just
the
read
file
sync,
which
opens
the
file
my
span
of
read
PNG,
which
is
also
Gathering
the
data
and
doing
a
little
processing
on
it.
But
but
it's
also
reading
all
the
data
1.29
milliseconds.
So
this
file
system,
access
of
1.29
milliseconds,
the
overhead
of
tracing,
is
going
to
be
in
the
microseconds.
It's
it's
negligible!
B
B
If
you,
if
you
make
a
query
into
honeycomb,
that's
like
over
more
than
a
few
hours
of
data
behind
the
scenes,
we
are
spinning
up,
Lambda
functions
to
read
the
data
from
S3
for
the
database
and
we
are
paying
for
every
one
of
those
but
we're
paying
for
it,
because
we
want
your
query
to
come
back
in
a
second,
maybe
two
seconds.
B
So
we
need
it
to
be
incredibly
fast
and
we're
paying
for
every
CPU
cycle
for
every
millisecond.
Those
lambdas
are
running
so
we
optimize
the
heck
out
of
those
to
keep
this
affordable
and
we
do
that
with
tracing.
B
So
I
would
say
if
you
care
about
performance,
the
overhead
of
tracing
is
more
than
made
up
for
by
the
ability
to
see
what's
taking
up
time
in
your
system
and
where,
where
you're,
spending
that
okay,
so
that's
that's
my
opinion
on
that
for
this
program.
How
long
does
this
program
take?
It
takes
1.8
seconds.
Nobody
cares,
I
mean
nobody
cares
about
the
overhead
of
tracing.
Also
I,
don't
really
care
that
it
takes
a
second
I.
Think
the
node
startup
time
is
much
longer.
A
B
Okay,
yes,
so
that's
fun,
tracing
locally,
like
we're
doing
here,
where
we're
Gathering
all
the
traces
and
using
it
for
local
debugging.
You
absolutely
want
every
Trace
to
be
recorded
in
this
local
Jaeger.
B
As
soon
as
I've
done
Docker
compose
it
will
be
gone,
but
you
want
to
keep
every
Trace
in
production
I'm.
Looking
at
this
and
there's
like
86
bands,
If
This
Were,
a
batch
job
that
I'm
running
in
production.
Okay,
actually,
if
it
were
best
job
I'd,
probably
want
to
keep
everyone
anyway.
B
But
if
this
were
web
service
request
or
even
a
batch
job
that
runs
on
a
queue
and
I'm
doing
a
million
of
these
in
a
minute.
I,
absolutely
don't
want
to
keep
86
million
spans.
B
The
generation
of
them
is
less
of
a
problem
than
the
storage.
Just
the
storage
gets
expensive,
especially
if
you
want
them
to
be
searchable,
and
in
that
case
you
do
sampling,
you
can
do
sampling
in
multiple
ways.
You
can
decide
at
the
beginning
of
a
request.
B
Okay,
we're
not
going
to
keep
this
one
and
then
all
of
your
span,
dot,
span
creation
and
span.set
attributes.
Those
are
all
going
to
be.
No
Ops,
if
you
decide
at
the
beginning
that
we're
not
keeping
it
and
then
there's
there's
ways
to
decide
at
the
end
instead,
so
that
you
can
do
things
like
keep
all
the
errors.
The
trick
with
tracing
is
that
you
want
to
keep
all
of
a
trace
or
none
of
it.
So
you
do
want
to
be
consistent.
You
don't
want
like
randomly
a
couple.
B
Read
files
from
this
run
and
a
couple
read
files
from
that
run.
The
point
of
a
trace
is
that
it
tells
the
store,
a
story
and
it's
a
coherent
story.
So
in
production
you
would
keep
one
out
of
ten
one
out
of
a
hundred
if
it's
a
health
check
one
out
of
a
thousand
whole
traces.
B
A
Think
there
is
a
one
more
I
think
and
then
someone
asked
for
the
template
demo
place
called
repo
again,
but
luckily
Carlos
sent
it
in.
So
as
you
got
their
answer
to
that,
so
that's
perfect
and
then
there
was
Ludovic
with
the
question.
What
your
point
of
view
on
using
agent
or
gateway
to
collect
pricing
on
kubernetes.
B
Okay,
I
am
gonna,
go
with
ask
me
on
Twitter
or
something
because
that's
totally
like
out
of
scope
for
this
particular
topic,
you
can
use
an
agent
or
a
Gateway
I,
don't
well
one
I,
don't
really
care.
I
would
probably
pass
that
opinion
to
or
that
question
off
someone
else.
B
I
don't
have
strong
feelings
about
that.
One
and
I
want
to
focus
here
on,
like
the
local
tracing
and
the
the
how
to
add
tracing
to
a
node
app
in
this
case.
Thank
you
for
the
question.
Anyway.
Linkedin
user.
B
Okay,
we
saw
that
we
can
read
files.
We
saw
that
we
are
reading
the
same
file
over
and
over
bring
that
back.
Okay,
I
need
to
expand
this
shrink
this
and
we're
reading
tree
multiple
times.
B
So
my
next
question
that
I
have
for
my
own
app
is
okay.
Why
are
we
reading
bigger
tree?
What's
reaching
bigotry?
I
can
go
with
that
in
a
couple
ways
of
like
what
here's
my
read
PNG.
What
calls
that
I
can
ask
where
to
go.
Okay,
read
PNG
is
called
I,
read
image,
but
read
images
like
just
one.
This
looks
like
just
a
wrapper
function.
That's
not
interesting!
So,
while
in
profiling,
where
it
does
like
every
function,
call
it
would
create
a
level
for
this
I'm
not
going
to
this
isn't
meaningful.
B
Clearly
at
some
point,
I
used
it
to
print
a
bunch
of
buggy
stuff,
but
I'm
not
now.
So,
let's
go
outside
of
that.
This
is
something
I
could
do
either
outside
or
in
or
inside.
Okay
looks
like
we'd:
image
is
called
in
four
differences,
so
any
of
those
can
be
interesting,
and
so
I
don't
know
like
where
do
I
want
to
follow
this
tracing
group.
B
So
another
thing
that
I
can
do.
Oh
go.
A
Ahead,
there's
another
one
again
Carlo!
Thank
you!
So
much
do
you
have
any
techniques
for
a
bit
or
best
practices
to
keep
tracing
code
separated
from
the
actual
code,
implementation.
B
In
general,
if
you
have
Auto
instrumentation
you're,
not
even
going
to
see
it,
the
the
calls
to
the
file
system
just
happen
and
just
get
instrumented
that
one's
not
actually
called
or
it
would
show
up.
But
when
the
tracing
is
about
your
code
and
you're,
making
custom
spans,
don't
separate
it
from
your
code
now
I
can
say
that,
but
I
also
will
say:
do
wrap.
It
create
a
little
wrapper.
B
Where
did
I
put
that
app
span
here?
Okay,
so
like
here,
where
we're
doing
read,
pianchi
and
I
I
do
this
enough
times
that
I
have
an
alias
for
it.
B
I
say:
return
start
active
span,
and
so
that
part
the
need
to
type
in
each
time
and
the
name
should
be
searchable
in
your
code.
You
want
to
hard
code
that,
but
then
I
always
have
to
remember
to
close
the
span
before
the
return.
Also
I've
left
out
the
part
in
these,
where
the
part
that
I
put
in
main,
where,
if
there's
an
exception,
I
want
to
catch
it
and
put
on
the
span
and
the
span
anyway,
so
I
actually
would
recommend
making
yourself
a
little
wrapper
function.
B
That
does
what
you
want
and
calling
that
instead
of
the
open,
Telemetry
start
active
span
and
that's
that's
gonna
be
I,
don't
know
it
might
be
10
lines,
but
that's
that's
as
far
as
I
would
go
with
obstruction.
This
part
set
attribute
put
this
in
your
code.
B
If
you
don't
have
the
S
variable,
you
can
also
do
this
with
otel.trace.getactive
span.
You
can
do
that
anywhere.
Even
if
you
don't
know
where
you
are,
for
instance,
I
could
do
that
in
Main
in
Maine
in
a
function,
Maine
doesn't
know
about
the
tracing
around
it,
and
this
Trace
ID
is
actually
really
important.
B
So
when
we
we
put
it
to
the
console,
so
I
would
do
hotel,
Trace,
dot,
get
active
span,
question
mark
in
case
it's
null
set,
attributes
app
dot,
created,
Trace
ID
and
put
this
in
here
now.
A
lot
of
people
worry
about.
This
is
cluttering
up
my
code.
B
Instead,
if
you
have
tracing
with
console.log
this,
because
it's
important
and
adding
a
set
attribute
to
your
code
says
something
about
what's
important
in
that
function
and
I
find
it
actually
expressive,
I
mean,
maybe
you
want
a
little
wrapper
function,
for
this
wrapper
functions
are
great
because
they
let
you
like,
create
little
organizational
standards
such
as,
maybe,
instead
of
app
you
add
a
different
prefix
to
every
or
just
a
consistent
prefix
to
every
field.
B
Name,
so
do
create
wrapper
functions,
but
don't
try
to
like
completely
separate
the
tracing
from
your
code,
make
this
searchable
between
the
attribute
on
the
Span
in
your
tracing
system
and
the
code
itself,
because
that
draws
a
connection,
and
this
is
how
you
get
back
and
forth
from
your
code
to
okay.
Where
is
this
in
my
traces?
You
can
search
for
that
field
and
from
the
trace
to
okay.
I
see
that
there
was
an
error
on
this
span.
What
code
is
that
cool
okay?
Carlos
says
it
makes
sense,
sweet.
B
A
B
A
B
Okay,
but
for
here,
let's
see
how
we
got
as
far
as
as
finding
that
there
are
all
these
read:
pngs
of
duplicate
files,
The
Next
Step.
The
take
here
is
to
find
out
what
those
are
part
of
either
by
going
up
or
by
going
down.
In
this
case,
it
turns
out
to
be
useful
in
a
lot
of
cases
to
also
like,
as
I'm
moving
up
from
the
file
I
also
go
down,
because
I
want
to
find
like
what's
an
action.
What
are
all
these
file
reads?
B
Part
of
and
my
main
function.
I
have
a
couple
main
steps.
Read
image?
Oh
well,
we
know
what
that's
doing.
We
already
have
a
record
for
that
plan
spans
and
send
spans,
and
so,
if
I
go
into
like
plan
spans,
which
is
a
major
step,
it's
doing
a
couple
things,
but
I
could
span
at
a
span
for
this
plan
sends,
which
is
a
fairly
easy
one.
B
Oops
enter
down
down
down
down,
don't
forget
to
end
this
band
before
the
return,
and
this
will
give
us
a
span
for
just
like
one
level
down
in
Maine,
and
we
can
see
how
many
of
the
file
reads
are
inside
of
plan
spans,
and
then
we
can
kind
of
zero
in
on
the
areas
that
I
want
to
refactor
and
then
I
haven't
done
any
it's.
B
We
don't
have
to
get
this
done
in
an
hour,
then
I
can
try
to
shift
the
spans
and
the
look
of
them
as
I
as
I.
Do
a
refactor.
Okay,
let's
see
if
plan
span
showed
up
so
this
is.
This
is
another
thing
that
I
do
when
I'm
adding
tracing
is
I
just
every
time.
I
want
to
see
a
difference.
I
love
to
see
a
difference.
Some
people
love
seeing
green
tests
and
that
just
like
does
it
for
them
and
they
use
tdd
all
the
time
and
tdd
is
excellent.
B
It's
a
fantastic
tool,
it's
a
tool
for
thinking,
a
tool
for
design
and
it's
a
it's
a
great
way
to
code,
but
sometimes
you're
presented
with
a
program
that
doesn't
have
tests
and
you
can't
just
do
tdd
on
something
like
that:
tracing
works
for
both
and
I
love.
Seeing
a
difference
in
my
Trace
I
love.
My
software
telling
me
hey
check
this
out.
I
did
something
new,
oh
and
spans
right
here:
okay,
okay!
So
if
I
like
shrinks,
no,
no,
no!
No!
No!
No!
No
over
here
shrink;
no,
no!
No!
There!
B
Okay,
I!
Can
here's
our
read
PNG,
that's
at
the
beginning
of
May,
and
here
is
plansoms.
So
it's
all
of
this
incorporates
all
of
those
things
all
of
those
files
and
say:
okay.
Well,
it's
somewhere
in
plan
spans
and
then
I
can
dig
into
this
and
I'm
like
okay.
There
are
three
main
components
to
plan
spans:
I'm
gonna,
cheat
I
know
which
one
it
is
and
but
I
can
see
which
of
those
are
taking
time
which
of
those
are
reading
files.
B
Oh
watch
out
I
think
this
is.
This
is
the
hard
one?
No,
this
is
the
hard
one
and
then
I
need
to
get
this
to
the
end,
which
is
actually
tricky
where's.
The
end
of
this
function:
94.,
okay,.
A
And
then
some
person
is
asking
also
is
the
tool
you
mentioned
tdd
question
mark.
Oh.
B
Oh
okay,
great
question:
tdd
is
called
test
driven
development
and
error.
We
don't
make
a
change
to
the
code
until
we've
run
a
test.
You
start
with
a
failing
test
like
if
you
want
to
add
a
feature.
You
start
with
the
test:
the
tests
that
the
feature
works
and
the
feature
doesn't
work
because
you
haven't
added
it
yet
and
then
you
add
the
feature
and
then
the
test
is
green
and
this
is
really
satisfying
and
then
from
from
Green
tests.
B
When
you
have
everything
nicely
unit
tested
in
automated
tests
that
you
can
run
with
a
push
of
a
button
or
automatically
on
file
save,
then
you
can
refactor.
You
can
move
your
code
around
without
having
an
effect
on
the
outside
and
all
your
tests,
Stay
Green,
and
that's
really
satisfying
in
my
case.
I.
Don't
have
tests
to
stay
green,
so
I'm
looking
at
the
traces
do
they
do.
B
What
I
expect
and
this
this
is
teaching
me
about
how
the
program
works
and
it's
something
that
I
can
build
in
gradually,
which
is
harder
to
do
with
tests.
B
And
I
don't
have
a
tracer,
so
we
need
to
do
that
so
as
I
as
I
like
add
additional
spans
around
things
that
are
important,
I
get
a
better
idea
of
like
the
shape
of
the
program
and
not
just
the
code,
but
the
shape
of
the
program
execution
like
with
these
operations.
I'd
plan
spans,
which
takes
a
long
time
which,
which
is
hard
Diego,
should
join
me.
This
now
find
traces.
B
I
want
like
one
more
span:
oh
88
spans
that
look
good,
it's
random!
How
many
pictures
it
builds
in
to
the
trace,
and
here
oh
I,
think
someone
asked
where
they
can
learn
about
tdd.
Oh,
let
me
give
you
my
favorite
recommendation.
B
Gpawhill.Org
gpaw
is
the
best
for
teaching
about
PDD
sheepile.org.
B
Okay,
my
router
is
not
making
new
connections
there.
It
is
yeah
highly
recommend,
here's
the
great
material
for
working
in
smaller
increments
and
generally
a
great
programmer.
B
Oh
he's
he's
funny
and
very
real
okay
Planet
spans
we've.
We've
got
to
read,
build
pictures
in
waterfall,
so
build
pictures
and
fall
now
most
of
the
read
files
under
it.
So
I
know
that
it's
part
of
build
pictures
in
waterfall
that
it's
reading
the
same
file
over
and
over
and
at
that
point
I
can
start
moving.
Those
reads
out.
B
Which
I'm
not
sure
I
have
time
for
today,
because
Annie?
Is
it
okay?
What
if
I,
if
I,
show
what
what
these?
What
this
Trace
looks
like
in
honeycomb,
with
its
little.
B
Fine
check
this
out
I'm
also
sending
these
to
Honeycomb
with
my
local
collector
and
yeah.
Oh,
it
looks
like
I
have
two
of
them
going
on.
So
let
me
check
my
recent
traces
and,
let's
get
now,
you
stop
Group
by
Trace
ID
last
10
minutes
there
we
go
okay,
so
here
is
my
trace
of
everything
where
I
can
look
at
the
file
name.
B
B
And
here
is
yep
here
it
is
here's
the
the
trees
and
the
balls
and
I
need
to
zoom
out
to
make
it
look
there.
We
go
and
the
little
song
and
some
sort
of
colors
here,
that's
okay.
What
I
really
wanted
was
this
Trace
ID?
B
Let
me
go
to
this
board,
so
I
get
just
one
of
them:
okay,
and
really
what
so
now
I'm
I'm
graphing
the
spins
in
this
one
trace-
and
here
they
are-
is
a
stacked
graph,
and
so
it
makes
a
tree
in
this
case,
and
here
in
the
heat
map,
it
makes
the
whole
drawing
so
this
is
this:
is
the
game
of
let's
switch
to
Absolute
time?
For
me
of
this,
the
program
is
that
it
draws
a
picture
in
a
heat
map.
B
B
B
The
the
program
that
I'm
working
to
make
configurable
turns
a
PNG
it's
reading,
pngs
and
making
a
picture
in
which
is
some
visualization
art
that
I'm
rather
happy
with.
And
then
you
can
do
things
like,
say:
hey,
what's
different
about
these
spans
and
it
turns
out
that
what's
different
is
their
reindeer
name,
Dozer
dancer,
pixels
and
then
you
can
group
a
group
by
the
reindeer
name
and
oh,
no,
no
I
need
I
need,
put
it
back
in
the
time
range
custom.
B
B
You
can
already
draw
your
any
PNG,
but
but
regardless,
when
you
click
into
a
Trace
you're,
always
going
to
see
The
Christmas,
Song
and
you're
always
going
to
see
ornaments
and
trees
and
after
I
move
move,
ornaments
and
tree
reading
up
to
the
beginning,
then
we'll
read
the
files
less,
but
also
it'll
I'll
be
able
to
accept
a
configuration
file
that
says
which
drawing
should
be
in
here.
All
these
drawings
are
configured
in
here
in
no
no
here's,
here's
the
bigger
tree.
B
It
looks
like
us
in
a
PNG,
and
this
is
the
PNG
that
gets
converted
into
Santa
and
his
reindeer.
B
So
that's
that's
the
fun
yeah,
yeah
and
and
the
actually
in
here
it's
a
little
complicated,
but
the
the
blue
channel
is
like,
like
how
dark
to
be
in
the
heat
map
and
the
red
Channel
correspond
to
the
Red
Dot
Json
to
the
attributes
on
the
span.
A
Yeah
perfect
such
a
great
image,
lovely.
A
We
have
four
minutes
left,
so
if
anyone
has
any
questions,
I
highly
recommend
you
to
type
them
out
now.
So
you
have
time
to
get
to
a
few
of
them
at
least
but
yeah.
Anything
else
that
you
want
to
do.
A
Oh
there's
a
question
already:
will
it
be
possible
for
you
to
show
the
RG
R
in
your
code
implementation?
What's
our
GR?
A
Yeah
anything
else
that
you
wanted
to
show
us
or.
B
I
yeah
yeah
I
didn't
quite
get
as
far
as
like
doing
a
refactor
and
seeing
the
trace
change,
but
I
did
earlier
today
when
I
practiced
this
and
it
it's
really
satisfying
and
I.
Think
that
this
oh
red,
green
refractor
for
tdd,
would
it
be
possible
for
you
to
show
read
Green
River
in
your
code
implementation?
Definitely
not
because
I
don't
have
any
tests
don't
have
a
test
framework
set
up,
did
not
write
this
for
testability
I
wrote
this
for
hurry.
B
Hurry,
marketing
has
a
thing
going
on
and
I
want
it
to
be
cute
and
it
has
to
be
out
before
Christmas.
So,
there's
like
this
Christmas
deadline
in
general,
I
think
T,
red
green
refactor
and
test
driven
development
or
as
part
of
test
driven
development
is
a
great
great
way
to
code,
especially
when
you,
you
intend
the
code
to
go
into
production
which
this
was
never
intended
for,
but
often
you
have
code
that
was
not
intended
to
go
into
production,
but
now
they're
like
ship
it.
B
The
customer
service
reps,
are
like
oh
we'd
love
to
put
this
in
our
newsletter
and
send
it
to
the
customers
and
but
I'm.
Like
but
it's
not
Christmasy,
it's
due
Christmas
Eve,
it's
too
hard-coded,
and
so
when
you
want
to
move
a
piece
of
code
from
well,
it
works,
but
I
can't
easily
change
it
because
I
don't
know
when
I'll
break
it
right,
it's
too
fragile
because
it
doesn't
have
tests.
B
B
For
instance,
if
I
get
a
full
configuration
file,
including
the
random
seed
and
the
time
of
day,
that
will
deterministically
produce
a
set
of
spans
for
the
output
that
that
draw
the
three
pictures
and
that
I
could
put
a
test
around
and
then
I
can
put
a
test
around
the
individual
Parts,
because
all
the
file
reads
are
happening
at
the
beginning
and
all
of
the
the
open
Telemetry
send
spans
are
happening
at
the
end
and
that
that
will
make
each
part
in
the
middle
testable,
and
so
this
is.
B
This
is
moving
the
code
to
production,
ready,
testability
wise.
At
the
same
time,
it's
immediately
making
it
more
production
ready
by
giving
you
visibility
into.
What's
happening
at
runtime,
because
these
same
traces
that
are
helping
me
it,
while
I'm
coding
will
help
in
production
at
a
smaller
scale,
we'll
sample
them.
Then.
A
Perfect
and
then
Daniel
asked
compliment
that
this
is
interesting
and
then
wanted
to
know
where,
where
to
learn
more
about
observability
and
also
ask
to
a
recording
of
this
be
available,
so
I
can
answer
that
it
will
be
available
and
it's
going
to
be
in
the
Cloud
native
Computing
foundation
and
YouTube
I
think
in
just
a
matter
of
an
hour
or
so
so
you
can
check
out
the
recording
there
very
soon,
but
as
the
final
question
I
guess
for
today
and
after
that
we'll
wrap
up.
A
What
do
you
think
is
the
best
ways
for
people
to
learn
about
these
Concepts
as
a
Next
Step?
Oh.
B
Well,
I
say
that
the
cncf
has
a
couple
other
open,
Telemetry,
specific
meetups,
there's
open,
Telemetry
and
practice.
There's
an
open,
Telemetry
and
user
group,
end
user
discussion,
group
I
think
and
open
telemetries,
the
Modern
Standard
for
observability.
So
that's
a
great
place
to
come
and
talk
to
people
and
learn
more.
A
Perfect,
so
that's
where
everyone
will
be
headed
after
this
always
great
to
have
the
next
few
stories
ready,
but
thank
you,
everyone
for
joining
the
latest
episode
of
cloud
native
live.
It
was
great
to
have
a
session
about
open
Telemetry
today,
and
we
also
really
love
the
interaction
as
well
as
the
question
from
the
audience
amazing
questions.
Amazing
reactions
always
very
loudly
to
hear
that
and,
as
always,
we'll
bring
you
the
latest
Cloud
native
code,
every
Wednesday
and
in
the
coming
weeks
we
have
more
great
sessions
coming
up
so
tune
in
for
those.