►
From YouTube: OpenTelemetry Auto-Instrumentation Deep Dive - Carlos Alberto Cortez & Alex Boten, LightStep
Description
Don’t miss out! Join us at our upcoming events: EnvoyCon Virtual on October 15 and KubeCon + CloudNativeCon North America 2020 Virtual from November 17-20. Learn more at https://kubecon.io. The conferences feature presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
OpenTelemetry Auto-Instrumentation Deep Dive - Carlos Alberto Cortez & Alex Boten, LightStep
Auto-instrumentation allows users to monitor their applications without the need to modify the code base, and immediately start gathering observability data . As part of the OpenTelemetry initiative (resulting from the merge of OpenTracing and OpenCensus), auto-instrumentation libraries will become a core feature, and will be offered across different languages (Java, Python, Ruby, Node.js, .Net, etc). In this deep dive you will learn about the architecture of these auto-instrumentation libraries, out-of-the-box OSS libraries integrations (such as Spring, Django or Rails); how to configure them to export telemetry data to different tracing and metrics backends (such as Jaeger or Prometheus), as well as interesting challenges, such as the possibility to share OSS integrations between auto and manual instrumentation.
https://sched.co/Zein
A
A
We're
very
happy
about
having
the
opportunity
to
keep
this
presentation.
Even
if
it's
a
little
bit
delayed,
you
know,
but
what
they
say
it's
better
later
than
ever.
So
thanks
so
much
for
joining,
definitely
yeah.
I
think
we
will
be
going
through
the
slides.
So
the
first
thing
is
that
we
were
actually
not
very
sure
about
the
public
and
how
many
of
you
watching
this
presentation
are
familiar
with
open
telemetry,
so
we
are
going
to
give
a
very
small
introduction.
A
So,
first
of
all,
open
telemetry
is
a
cncf,
but
baked
project
is
an
open
source
project.
The
idea
behind
this
project
is
to
provide
a
complete
rich
environment
to
provide
and
capture
telemetry
data
as
part
of
your
services
and
then,
of
course,
provide
this
information
as
distributed
traces
and
metrics
and
other
signals
in
the
future,
and
the
data
is
that,
once
you
have
all
these
telemetry
data
out
of
your
service,
this
can
be
sent
to
the
backend
of
your
choice
and,
of
course,
to
make
this
actually
a
reality.
A
We,
the
idea,
is
that
we
will
be
providing
a
complete
ecosystem
for
many
languages,
which
includes
go
java,
chart
python,
javascript
and
so
on.
So
first
of
all,
of
course,
there's
an
api
which
will
allow
people
instrumenting
their
applications
or
their
libraries.
You
know
just
go
and
say
hey.
I
want
to
send
this,
and
I
want
to
send
that
at
this
spot,
not
only
that
we
are
also
providing
an
actual
sdk
implement
implementation
that
you
can
think
of
it
as
a
reference
implementation.
A
Part
of
this
is
that
we
also
want
to
provide
out
of
the
box
instrumentation
for
popular
frameworks
like
django
for
python
or
spring
for
java.
There's
one
more
component
called
a
collector.
The
collector
is
a
component
that
actually
we
can
see
it
as
a
sidecar.
It
can
receive
this
telemetry
data,
massage
it
process
it
normalize
it
and
then
finally
export
all
these
telemetry
data
to
the
backend
of
your
choice
and,
of
course,
instrumentation,
which
is
the
one
that
you
guys
are
most
interested
in.
A
B
B
Why
and
one
of
the
one
of
the
key
benefits
of
auto
instrumentation
is
to
help
folks
that
are
already
busy
already
overworked,
find
a
way
to
instrument
their
applications
more
easily
and
if,
if
you're
not
familiar
with
distributed
tracing
or
with
metrics,
it
could
be
a
huge
barrier
to
cross,
and
so
one
of
the
benefits
of
auto
instrumentation
is
that
you
can,
with
with
a
fairly
simple
installation,
step
and
a
script
or
a
different
command
to
run
your
application
with.
You
can
start
collecting,
telemetry
information
for
your
application.
B
One
of
the
other
benefits
with
auto
instrumentation
is
that,
with
zero
or
very
little
code,
you
can
provide
insights
to
users
of
particular
libraries
as
to
how
those
applications
how
their
applications
are
leveraging
those
libraries
and
they,
the
libraries,
will
provide
rich
metadata.
That
is
important
to
the
users
of
those
libraries
and
so,
as
you'll
see
we'll
be
diving
into
the
demo.
B
Today,
we're
going
to
show
you
a
compo,
a
system
composed
of
multiple
applications,
some
in
java,
some
in
python
that
can
be
instrumented
with
little
or
no
code
and
then
we'll
dive
a
little
deeper
to
talk
about
how
the
implementation
is
actually
done
in
python
and
in
java
I
I
love
having
a
good
picture.
So
here's
a
little
bit
of
what
the
application
looks
like.
B
So
there
is
a
client
here
that
will
make
a
request.
The
client
is
written
in
python.
It
will
make
a
request
to
a
server
written
in
python
as
well.
The
client
is
instrumented
will
be
auto
intermittent
via
the
request
library.
The
server
is
using
flask.
That
server
makes
a
request
to
a
sql
server
that
just
has
a
list
of
work
from
home
tips.
A
B
Anybody
who's
looking
for
more
tips
that
flash
server
will
then
make
a
request
upstream
to
a
so
another
server
written
in
java
using
spring,
and
that
server
will
return
a
count
of
how
often
a
particular
tip
has
been
blogged
about
in
the
past
five
months.
Important
note
here,
all
of
the
data
is
static
and
has
absolutely
zero
scientific
value
and
each
one
of
these
services
is
actually
going
to
be
using
a
jaeger
backend
for
collecting
inflammatory
information.
A
Yeah
fantastic
now
is
the
time
for,
for
you
guys
to
check
out
what
this
looks
like
in
real
life,
so
I
will
go
through
so
essentially
you
know
these
components
that
our
services
that
alex
mentioned,
I
will
go
through
the
code
quickly,
so
we
have
first
the
the
flash
server,
which
is
just
as
you
can
see.
This
is
super
something
very
simple.
You
just
make
a
call
somewhere
else
in
this
case
the
actual
java
code.
A
Then
we
have
the
request,
client,
which
is
something
oh.
This
is
actually
what
I
was
showing.
So
the
flask
server
is
the
code.
You
can
see
that
this
uses
split
out
from
me
inside.
You
know
as
part
of
the
code
base
and
everything
and
nothing
fancy.
We
are
not
sending
any
traces
of
part
of
these
we're
just
doing
our
thing.
And
finally,
I
will
show
also
the
spring
service
springs.
A
Second,
and
then
we
are
just
starting
the
application
and
doing
the
usual.
You
know
just
making
a
call
a
pair
of
clothes
to
raise,
there's
nothing
related
or
specific,
to
tracing
or
to
open
to
elementary
now,
for
the
first
run,
what
they
will
be
doing
is
that
I'm
going
to
start,
I'm
gonna
run
these
three
services,
but
only
one
of
them
is
gonna
be
automatically
instrumented,
so
you
can
feel
the
actual
difference.
A
A
A
So
first
we
have
the
flag
server,
which
had,
as
I
said,
nothing
related
to
tracers
or
to
open
celebrities
with
the
code
base,
and
it's
already
you
know
creating
this
trace
where
you
can
see
how
much
time
was
spent
on
on
the
sql
query,
and
you
can
see
also
extra
information
like
what's
the
command
and
other
stuff
for
the
flash
server
itself
you
can
see.
What's
what
was
the
method?
A
A
But
of
course
we
have
to
also
go,
and
now
I'm
going
to
change
things,
so
we
are
going
to
run
the
client
without
instrumentation.
Now
I'm
going
to
use
a
very
small
script.
Sorry
so,
as
you
can
see
this
script,
the
only
thing
that
does
is
that
it
just
configures.
What's
the
end
point
for
jager,
which
is
the
back
end
of
your
choice
and
also
configuring
the
sdk.
As
I
said,
we
are
using
this
preference
sdk.
A
A
This
is
the
entire
thing
and
of
course,
it
also
has
additional
information
like
components,
the
method,
status
code
and
so
on,
and-
and
all
of
this
was
done
without
touching
your
actual
code,
which
can
be
pretty
useful.
You
know
like
just
imagine
that
you
want
to
try
out.
What's
all
open,
telemetry
phone
is
about
and
say,
let's
give
me
you
know,
let's
give
it
a
run.
You
just
need
to
configure
your
back
end
and
a
pair
of
things
and
that's
about
it.
A
Thank
you
and
now
going
forward
now.
How
does
it
work?
I
guess
that
this
is
the
obvious
question
right.
This
is
part
of
an
actual
deep
value,
so
let's
go
through
the
actual
phone.
So,
first
of
all,
usually
the
way
you
actually
instrument
your
job
application
is
that
we
use
the
java
agent
facilities.
A
A
In
turn,
this
agent
contains
a
few
components.
The
first
of
all,
the
first
one
is
the
auto
instrumentation
engine,
which
is
the
one
that
will
launch
your
application
and
will
eject
code
to
make
this
instrumentation
possible,
but
we
also
need
actual
instrumentation
for
popular
libraries
like
spring
your
pc,
ok,
http
and
so
on.
We
have
a
lot
of
instrumented
libraries
these
days,
which
is
going
to
help
users.
You
know
they
can
forget
about
trying
to
actually
instrument
those
super
popular
libraries
and
then
by
default.
A
We
usually
try
to
use.
We
try
to
avoid
going
the
bytecode
manipulation
way.
We
are
trying
to
use
available
hooks
events
or
interceptors.
These
two
small
screenshots
show
how
it's
done
there
for
both
in
those
cases
for
all
http
and
jrpc
we're
using
simply
interceptors,
which
is
very
simple.
You
know,
and
it's
very
nice
you
don't
have
to
actually
manipulate
anything
and
by
body
which
is
one
that
we
use.
A
We
need
to
use-
hopefully
not
so
much
to
actually
inject
these
instrumentation
libraries
at
runtime
when
your
application
is
starting
when
it
comes
to
the
actual,
how
it
looks
instrumenting
a
library.
There
is
an
instrumental
interface
that
everybody
who
wants
to
instrument
their
application
need
to
extend.
The
good
thing
is
that
there
is
a
default
implementation
that
goes
for
the
most
common
cases
and
then,
of
course,
in
this
example,
you
can
see
here.
A
A
And
then
running
the
application
for
java.
It's
on
you
know,
as
I
said,
the
engine
itself
is
simply
a
jar.
It's
a
single
file
and
then
you
have
to
provide
the
configuration
yourself.
Usually
that's
done
through
environment
variables,
which
is
a
very
common
practice,
but
you
can
also
specify
them
as
something
which
is
specific
to
java
called
system
properties,
which
is
basically
parameters
that
can
be
passed
directly
to
the
to
the
java
runtime.
A
B
Yeah,
so
in
on
the
python
side,
if
you
want
to
get
started
with
auto
instrumentation,
the
first
thing,
the
first
thing
we
want
to
do
is
go
and
install
the
open,
telemetry
instrument,
instrumentation
package,
which
you
can
do
through
pip.
This
package
contains
a
few
different
components:
it
has
the
open,
telemetry
bootstrap
and
the
open
telemetry
instrument
script,
the
instrument
interface-
that
will
describe
a
little
bit
further
shortly
and
then
the
sitecustomize.pi
file
for
open,
telemetry,
instrumentation.
B
So
first
things.
First,
the
open,
telemetry
bootstrap
script
is
a
nice
convenience
script
that
allows
users
to
to
discover
what
auto,
instrumentation
or
instrumentation
libraries
exist
for
their
environments,
and
it
does
this
by
doing
a
lookup
with
pip
of
all
the
installed
libraries
in
an
environment,
and
then
it
compares
that
list
against
a
list
of
known
instrumentation
libraries
and
it
prints
out
the
recommendations.
So,
for
example,
in
this
case
here,
it
recommended
installing
flask
jinja,
2
and
sql
alchemy.
So
this
is
after
running
it
for
the
in
the
server
environment.
B
There's
also
a
install
flag
for
open
telemetry
bootstrap,
which
allows
the
user
to
just
let
the
open
telemetry
boost
apps
script.
Does
the
installation
for
them
and
it
does
so
via
pip,
so
what
is
an
instrumentation
library
in
python?
It
really
has
to
do
two
things.
One
of
the
responsibilities
of
the
instrumentation
library
is
to
implement
an
instrumental
interface,
much
like
in
in
java.
This
interface,
as
you
can
see
here,
on
the
right
hand,
side
is
fairly
minimal.
It
has
an
instrument
and
an
instrument
method.
B
I'll
touch
base
a
little
bit
further
on
the
instrument
method
shortly.
But
it's
important
to
note
that
the
instrument
and
instrument
methods
are
implemented
is
library
specific.
As
far
as
their
implementation
goes,
some
libraries
provide
hooks
that
can
be
configured
while
others
don't
have
that
nice
facility.
So
we
end
up
using
monkey
patching
for
a
lot
of
those
and
for
those
who
aren't
familiar
with
monkey
patching
it.
It
just
means
that
we're
overriding
the
call
to
a
specific
library,
library
method
call.
B
B
Other
python
applications
within
a
system,
and
so
the
instrumentation
library
must
register
its
entry
point
under
the
open,
telemetry
instrumenter
entry
point,
and
so
here
you
can
see
the
the
flask
entry
point
is
just
the
flask
instrumenter
and
it's
defined
inside
the
setup.pi
file,
whereas
the
request
instrumenter
is
defined
within
a
setup.cfg
file
and
we'll
go
further.
So
why
we
need
these
entry
points
shortly.
B
But
the
next
component
that's
installed
is
the
open
telemetry
instrument,
which
is
used
to
run
the
the
application
itself
and
the
reason
why
we
need
to
to
use
the
open,
telemetry
instrument
script
to
run.
The
application
is
because
we
want
to
ensure
that
the
instrumentation
has
a
chance
to
be
applied
before
anything
in
the
application
is
done
so
that
we
don't
miss
any
any
call
from
the
application
itself,
and
it
does
so
by
using
the
site
customize
file,
which
looks
a
little
bit
like
this.
B
There
is
a
couple
of
gotchas
that
I'd
like
to
bring
up
to
everybody's
attention
because
they're
not
super
obvious
when
you
get
started
with
open,
telemetry,
auto
instrumentation
for
python-
and
I
think
carlos
touched
on
this
in
the
demo.
But
there
are
a
couple
of
things
that
you
want
to
configure
ahead
of
time,
and
these
are
things
like
the
meter
provider
and
the
tracer
provider
by
default.
B
The
open
telemetry
api
by
design
implements
a
no
op
tracer
provider
and
a
no-op
meter
provider,
but
more
often
than
not,
you'll
want
to
configure
this
to
use
the
sdk
provider
unless
you're
implementing
your
own
sdk,
for
example.
But
the
other
thing
that's
a
bit
of
a
gotcha
here
is
that
not
everything
that
you
would
like
to
have
configured
is
available
for
enviro
via
environments
variables.
Yet
in
python
there
is
active
work
happening
even
today
on
this
to
make
it
better.
B
B
B
B
And
you
know,
as
as
you
can
imagine,
there's
gazillion
libraries
out
there
for
all
of
the
different
languages,
and
so
it
it's
going
to
take
a
lot
more
work
to
ensure
that
we
have
coverage
for
all
of
the
libraries
that
are
important
to
people
and,
if
you're
looking
for
a
way
waiting
involved.
This
is
actually
a
really
good
way
to
to
get
involved.
If
you
have
a
favorite
library
that
isn't
currently
supported,
feel
free
to
open
an
issue
to
this
sig
of
the
language
of
your
choice.
B
If
you'd
like
to
give
a
try
to
the
existing
libraries-
and
you
know,
report
any
issues
or
any
missing
documentation,
that
would
be
another
good
way
to
get
involved.
And,
of
course,
if
you
have
time
to
write
some
comfort
for
us,
that
would
be
great.
We
were
always
excited
to
cprs
from
from
our
users,
so,
okay
yeah.
So
if
you're,
if
you're
looking
for
more
information
about
open
telemetry,
I
highly
recommend
checking
out
the
website.
B
It
has
all
of
the
links
and
info
about
the
different
sigs
different
languages
links
to
notes
of
the
of
the
cigs,
and
it
also
has
a
super
useful
registry.
So,
if
you're
looking
for
any
instrumentation
or
any
languages
that
you're
interested
in
you
can
check
out
the
registry
and
just
search
search
it
to
see
if
your
library
is
already
supported
and,
of
course,
if
you're
interested
in
the
demo
that
carlos
showed
earlier
here
is
the
link
to
where
you
can
find
the
code
and
yeah.
So
that's
that's
about
it.
B
If
you're,
if
you're
looking
for
more
information
about
open
telemetry
or
if
you
have
any
questions,
you
can
feel
free
to
reach
out
to
carlos,
and
I
we're
always
always
available,
you
can
jump
into
getter
as
well.
If
you're,
if
you're
looking
for
more
information
there,
we're
always
happy
to
talk
to
the
folks
about
instrumentation,
about
open,
telemetry
or
or
anything
really
and
yeah
come
and
come,
and
talk
to
us
join
the
project,
we're
we're
currently
in
almost
j
phase.
So
it's
pretty
it's
pretty
exciting
time
to
get
to
get
involved.
A
B
A
Yeah,
so
basically
this
answer
is,
you
know.
It
includes
two
factors.
The
first
one
is
that
any
instrumentation
that
you
can
set
up
manually
has
a
cost,
of
course
like.
Even
if
you
do
that
without
the
agent
for
those
parts,
we
are
being
very
aggressive
when
it
comes
to
optim
optimizations,
so
in
general,
it's
it
has
like
a
very
decent
cost
now
the
usual,
so
the
other
part
is
that
you're,
usually
using
an
auto,
auto
instrumentation
component,
is
a
sometimes
you
get
a
little
bit
more
of
time
at
startup.
A
So
at
least
we
know
that,
of
course,
there's
a
cost,
but
they
were,
you
know
based
on
production
code,
which
is
great
and,
for
example,
when
it
comes
to
compare.
For
example,
I
can
tell
you
for
the
java
case,
specifically
the
agent
at
the
open
telemetry,
that
I
said
it's
based
on
the
data
doc.
One
is
it's
like
two
or
three
times
faster
at
startup
than
the
special
agent,
which
is
the
one
that
was
created
as
part
of
open
tracing.
You
know
open
tracing
and
open
sensors.
A
A
B
A
A
B
The
next
one
is
a
question
from
kevin.
He
asks.
Is
there
a
way
to
add
custom
tags
for
auto
instrumented
network
calls?
Yes,
you
can,
you
can
basically
add
you
can
add
any
number
of
metadata.
You
want
to
any
calls
that
are
happening
because
you
can,
you
can
basically
add
traces
to
auto
instrumented
applications.
We
didn't.
We
didn't
show
it
here,
because
we
wanted
to
just
show
that
with
very
little
or
no
tracing
code,
you
can
get
tracing
information
about
your
applications
using
auto
instrumentation.
But
you
can
definitely
you
can
instrument.
B
A
B
Yeah
and
just
to
add
to
that
there
there
is
some
work
underway,
at
least
on
the
python
side,
for
the
grpc
library,
instrumentation
library,
to
add
metrics
as
well.
So
we
didn't
get
a
chance
to
demo
it
because
it
wasn't
ready
at
the
time
of
the
recording,
but
that's
something
we.
We
definitely
want
to
spend
a
bit
more
time
in
the
future.
A
Yes,
so,
as
I
mentioned
before,
yeah
that's
something
that's
on
the
works
for
ga.
It's
gonna
be
an
experimental
feature,
because
we
know
there's
only
so
much
time
that
we
can
work
on,
but
yeah
we
will.
There
will
be
something
as
part
of
ga
and,
of
course,
eventually
we
will
have
a
more
stable
update
to
that
part.
B
A
Yes,
they
do
they
do.
We
don't
have
anything
specialized
for
kotlin,
but
if
you
think
there's
something
that
we
should
support,
please
just
contact
us.
I
think
there
was.
There
has
been
interest
in
adding
stuff
specifically
for
scala,
for
example,
and
yeah,
but
yeah.
Usually
it's
just
like
you
know
every.
If
it
works
with
java,
it
will
work
with
kotlin
or
it
should,
and
if
not,
we
can
have
it.
B
B
Yeah,
I'm
I'm
not
sure
if
it
would
be
a
sidecar
as
much
as
the
node.js
containerize
application
itself
would
be
instrumented
with
open
telemetry
via
the
javascript
auto
instrumentation
tooling.
It's
probably
worth
taking
a
look
at
the
open,
telemetry
js
repo
to
get
an
idea
for
how
auto
instrumentation
for
javascript
works.
A
Well,
it
shouldn't
there
are
they're
having
actually
security
concerns
on
the
api
on
on,
because
we
had
this
idea
of,
for
example,
doing
context
propagation
out
of
the
box,
even
if
the
user
didn't
configure
that.
So
we
are
working
hard
on
solving
those
issues,
but
those
things
wouldn't
be
happening
as
part
of
the
instrumentation.
They
are
just
part
of
the
api,
sdk
and
instrumentation
in
general,
but
yeah
we're
taking
this
very
seriously-
and
this
is
you
know,
being
part
of
the
cncf-
is
part
of
this.
B
Yeah-
and
I
think
I
mean
I
think,
like
with
any
any
code
whatsoever,
there's
always
security
risks.
You
know,
I
think
I
think
it's
it's
really
up
to
the
library
in
instrumentation
library
writers,
to
ensure
that
no
data
is
leaked
that
shouldn't
be
leaked.
I
think
it's
also.
It's
also
up
to
the
the
folks
that
are
leveraging
those
libraries
to
look
at
the
data
that
gets
populated
in
their
tracing
information
in
the
tracing
backends
and
and
ensure
that
no
sensitive
data
is
being
exposed.
B
A
Not
that
I
know,
I
think,
there's
no
evidence
of
that.
In
any
case,
you
know
let's
say
that,
for
example,
there's
some
case
like
this
or
another
one
that
is
creating
too
much
too
much
telemetry.
That's
you
can
most
of
these
instrumentation
agents
already
support
blacklisting
something
you
don't
want
to
to
instrument
and
if
they
don't,
they
will
have
it
in
the
future
and
of
course
you
can
always
leverage
sampling
if
it's
if
it
becomes
a
problem
but
yeah.
But
I
am
a
little
bit
curious
about
that.
B
B
Okay,
probably
the
last
question,
because
we
only
have
one
minute
left
here
and
then
we'll
have
to
go
to
slack
from
mahmoud
asking.
Is
there
any
overhead
to
on
requests
and
how
do
we
configure
probability
of
capturing
traces?
So
there
is
the
sampling
api
or
the
sampling
api.
That's
configurable!
In
the
open,
telemetry
sdk!
B
That's
that's
likely
how
you
would
configure
the
probability
of
capturing
traces
and
as
far
as
overhead
on
requests
I
mean,
I
think,
like
with
with
everything,
there's
always
going
to
be
overhead,
as
you
add
more
code
in
the
path.
So
it's
just
a
matter
of
you
know
how
much
overhead
are.
Are
you
comfortable
with
on
on
the
requests
and
yeah
any
any
last
thoughts
on
that
question,
carlos.