►
From YouTube: OpenTelemetry Agent and Collector: Telemetry Built-in Into All S... Steve Flanders & Trask Stalnaker
Description
Don’t miss out! Join us at our upcoming events: EnvoyCon Virtual on October 15 and KubeCon + CloudNativeCon North America 2020 Virtual from November 17-20. Learn more at https://kubecon.io. The conferences feature presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
OpenTelemetry Agent and Collector: Telemetry Built-in Into All Software - Steve Flanders, Splunk & Trask Stalnaker, Microsoft
Effective observability requires high-quality telemetry. OpenTelemetry enables it by making telemetry a built-in feature of all software. Learn how easy it is to enable telemetry data collection for any app and enable observability scenarios by configuring OpenTelemetry collector. Help us build the future where all software exposes its telemetry in a unified way. Where app owners concentrate on building great software while getting insights into application behavior and reliability effortlessly. And see how the road to this future looks like. This talk will feature a demo of automatic java application instrumentation and collector configuration to upload high-quality telemetry to the backend of your choice.
https://sched.co/ZevT
A
Hello,
everyone
and
welcome
to
our
session
open
telemetry,
collector
telemetry,
built
into
all
software
thanks
for
joining
us
first
off
like
to
introduce
ourselves.
My
name
is
steve.
Flanders,
I'm
a
director
of
engineering
at
splunk,
I'm
also
an
open
telemetry,
collector
approver
been
working
in
the
observability
and
monitoring
space
for
the
last
decade
and
was
actually
contributing
to
the
open
census
project
prior
to
it.
Joining
with
open
tracing
to
form,
open,
telemetry
and
joining
me
is
trask.
B
Hey
I'm
trask
stolenker,
I'm
a
software
engineer
at
microsoft,
I'm
a
maintainer
on
the
open,
telemetry
java,
instrumentation
project
and
I've
also
been
in
the
monitoring
business
sort
of
well
open
source
business.
At
least
for
a
decade.
I
wrote
and
maintained
open
source
java
monitoring
tool
called
glow
root
and
enjoy
that
space.
So
much
that
here
I
am
in
open,
telemetry
world
now.
A
Awesome
so
our
agenda
for
today
I
want
to
give
you
a
quick
introduction
to
what
the
open
symmetry
project
is
talk
a
bit
about
the
open
symmetry
collector
and
how
it
fits
into
the
observability
picture.
Then
do
the
same
thing
around
auto
instrumentation,
focusing
primarily
on
java
that
we
will
touch
on
other
languages
as
well
and
then
really
wrap
it
up
with
a
demo
showing
you
how
to
really
get
started
with
the
open
telemetry
project.
A
So
with
that,
what
is
open,
telemetry,
open
symmetry
is
a
cncf
project
and
it's
really
looking
to
solve
a
pretty
big
pain
point
in
the
observability
space.
If
you've
heard
of
observability,
you've
probably
heard
of
the
three
pillars
of
observability
being
the
different
data
sources
that
you
have
in
your
environment,
traces,
metrics
and
logs
well,
for
each
of
these
different
data
sources,
there
are
actually
different
layers
to
it.
You
have
the
api.
A
You
have
the
implementation
of
that
api,
all
the
data
collection
aspects
and
then
things
like
wire
formats
interrupt
formats
and
the
like.
Some
of
these
things
are
actually
language
specific.
So
when
you
start
talking
about
instrumentation
apis,
they
have
to
be
written
for
each
individual
language,
where
the
agent
and
collector
mechanisms
have
different
deployment
methods,
depending
on
the
type
of
architecture
that
you're
you're
deployed
into
open.
Symmetry
is
really
looking
to
solve
all
of
these
aspects.
A
The
idea
is
everything
that
you
deploy
within
your
environment.
To
get
telemetry
data
out
open
symmetry
is
providing
a
completely
vendor
agnostic
solution
too.
To
get
that
telemetry
data
to
the
back
end
or
back
ends
of
your
choice.
Open
symmetry
actually
does
not
offer
a
back
end,
though.
That's
where
it
kind
of
draws
the
line,
anything
instrumentation
data
collection,
even
things
around
semantic
conventions,
are
all
included,
and
then
you
can
choose
either
an
open
source
or
commercial
backend
to
send
your
data
to
now.
A
The
open
symmetry
project
is
currently
in
beta
for
most
of
the
languages.
There
are
a
few
that
haven't
reached
beta,
yet
there's
been
an
extensive
focus
on
tracing
to
start,
so
it's
quite
mature.
In
fact,
the
specification
has
already
reached
its
first
stable
release.
The
metrics
one
is
actively
underway,
and
that
should
be
wrapping
up
later
on
this
year.
It
will
also
be
stable
by
the
time
we
get
to
ga
and
ga
is
planned
for
later
this
year.
So
a
lot
of
active
contributions
going
on
now
and
again.
A
A
So
let's
talk
about
how
you
would
deploy
open
symmetry
into
your
environment.
So
if
we
think
about
having
one
or
more
hosts
each
running
one
or
more
applications-
and
maybe
we
want
to
send
the
data
to
one
or
more
different
back
ends,
the
typical
recommended
approach
would
be
to
go
ahead
and
deploy
the
open
telemetry
collector
as
an
agent
on
each
of
your
hosts.
That
will
allow
you
to
actually
collect
host
metrics
automatically
and
also
provides
a
way
to
collect
tracing
information.
A
Now
that
tracing
information,
as
well
as
application
metrics,
can
come
out
of
open
symmetry,
client
libraries,
these
are
going
to
be
language
specific
and
out
of
the
box,
are
actually
configured
to
send
directly
to
the
open
symmetry
collector.
So
if
you
deploy
it
without
configuration,
you'll
actually
have
telemetry
data
being
emitted
for
the
auto
instrumentation
libraries,
which
is
pretty
cool
and
then,
in
addition,
you
can
also
deploy
the
open
symmetry
collector
as
a
standalone
service.
A
Now,
if
you
look
at
the
open
symmetry
collector
as
a
single
component
that
exists,
it's
actually
one
binary
that
can
be
deployed
in
a
variety
of
different
form
factors
depending
on
your
needs.
It's
really
looking
to
make
it
easy
and
provide
a
vendor
agnostic
way
to
collect
this
data,
so
it
actually
receives
many
open
standards
today,
like
jaeger,
prometheus
and
zipkin,
and
it
can
export
to
both
open
source
and
commercial
backends.
A
A
All
this
is
made
possible
through
the
collector
as
well.
Now,
why
would
you
want
to
actually
deploy
a
collector
like?
Why
not
just
have
the
client
library
send
this
data
directly
to
the
back
end,
the
idea
is
really
to
offload
the
responsibility
from
the
application
for
two
primary
reasons.
One
is
you
don't
want
to
have
to
implement
it
in
every
single
language
like
trying
to
keep
that
consistent
is
really
hard
and
even
the
implementations
are
going
to
change
because
they're
language
specific?
A
Second,
you
really
don't
want
to
introduce
too
much
overhead
to
the
application.
So
you
want
to
do
the
minimum
necessary
to
gather
the
data
that
you
need.
That's
why
we
recommend
deploying
the
open
symmetry
collector
as
an
agent.
You
can
actually
have
the
instrumentation
pass
off
this
data
quickly
and
then
have
the
agent
handle
this
in
a
unified
way
and
be
able
to
handle
things
like
buffering
and
retry
compression
encryption.
All
the
things
that
you
care
about
that
could
be
additional
overhead
to
your
actual
application.
A
So
really,
the
idea
here
is
time
to
value,
make
it
very
very
simple
offloading
of
responsibilities
and
really
providing
a
consistent
experience
end-to-end
now,
if
we
think
about
the
architecture
of
the
collector,
basically
it
has
a
notion
of
receivers.
How
you
get
data
in
this
is
both
push
and
pull
based.
As
I
mentioned,
it
already
supports
many
of
the
open
source
standards,
including
jager
and
prometheus.
A
It
has
a
notion
of
exporters,
which
is
how
you
get
data
out
and
again
it
supports
both
open
source,
as
well
as
vendor
commercial,
specific
backends.
It
has
a
notion
of
processors,
which
is
a
way
of
manipulating
the
data
as
it
passes
through
the
collector.
So
this
handles
things
like
batching
and
cued
retry
but,
as
I
mentioned,
also
pii,
redaction,
tail-based,
sampling
and
other
specific
scenarios.
A
Now,
what
you
end
up
doing
is
actually
defining
what
we
call
pipelines
in
the
collector.
So,
for
example,
I
can
say
that
I'm
receiving
traffic
over
otop
we've
been
going
through
a
batch
and
cued
retry
processor
and
I'm
exporting
in
jager.
As
I
mentioned
it
handles
that
translation
of
data
format
for
you
automatically,
you
could
then
define
a
second
pipeline
where
again,
maybe
I'm
receiving
from
otop,
which
is
open,
telemetry's
format
that
it
includes
by
default.
A
That's
what
the
auto
instrumentation
and
other
client
libraries
follow
today,
and
I
could
have
that
then
export
to
two
different
destinations,
otlp
and
in
this
case
prometheus.
So
there's
a
lot
of
flexibility
in
choice
via
configuration.
What's
really
nice
about
this,
is
you
don't
have
to
actually
go
make
code
modifications
to
take
advantage
of
these
capabilities?
It
all
happens
in
the
collector
itself.
A
A
Now
the
collector
actually
has
what's
known
as
the
core
repository
as
well
as
a
contrib
repository
in
core.
You
have
all
the
open
source
aspects
and,
as
you
can
see
here,
it's
pretty
broad
support
for
both
traces
and
metrics.
In
the
case
of
open
telemetry
logging
is
just
starting
to
pick
up,
so
eventually,
logging
will
be
added
as
well
to
both
client,
libraries
and
the
collector,
but
it's
still
early
for
that.
A
The
expectation
is
that
they'll
probably
be
a
beta
later
this
year
for
it
in
the
case
of
contrib,
these
are
more
community-based,
so
this
could
be
vendor-specific
logic
or
it
could
even
be
open
source
software.
That
only
applies
to
a
smaller
subset
of
the
the
user
base
of
open
telemetry,
but
it's
a
way
of
adding
things
that
are
outside
the
core
functionality
of
open
telemetry
as
a
whole.
Again,
you
can
see
pretty
rich
adoption
here.
In
fact,
in
general,
two
of
the
three
major
cloud
providers-
aws,
sorry,
azure
and
jcp-
are
already
instrumenting.
A
The
backend
services.
Aws
is
just
getting
involved
right
now,
as
well
with
the
project.
All
the
major
vendors
are
also
contributing
at
least
exporters,
if
not
other
aspects
to
the
project
and
you're,
also
seeing
broad
adoption
and
support
from
many
end
users
as
well
with
that
I'd
like
to
turn
it
over
to
trash,
and
he
can
tell
you
a
little
bit
more
about
java,
auto.
B
B
So
this
is
the
list
of
all
of
the
instrumentation
all
of
the
java
libraries
that
we
have
instrumentation
for
so
far,
and
so
with
auto
instrumentation,
it's
important
to
so.
If
you
are
using
these
particular
libraries,
we
will
instrument
those
at
runtime
via
the
java
agent
and
capture
the
calls
the
requests
coming
into
those
libraries,
the
calls
going
out
of
those
libraries
what
your
jdbc
sql
calls
look
like,
and
so
this
is
where
we
really
think
that
the
the
power
of
the
community
coming
together
can
grow.
B
B
So
one
question
you
might
be
asking
is,
you
know:
auto
instrumentation
is
great,
but
what,
if
I
want
custom
telemetry
also-
and
so
I
like
to
recommend
starting
with
just
auto
instrumentation
out
of
the
box-
try
it
out
that
way.
You
can
start.
You
know
seeing
telemetry
flow
to
your
back-end
monitoring
system,
and
then
you
can
start
to
layer
in
custom
telemetry
on
top
of
that
or
inside
of
that.
B
So
depending
on
the
complexity
of
your
needs,
there's
a
couple
of
different
ways
to
add:
add
custom
telemetry
into
your
application
as
well.
As
I
mean
the
the
for
the
open
telemetry
api
is
a
full-blown
telemetry
api
and
the
auto
instrumentation
auto
correlates
captures
and
autocorrelates
everything.
B
B
For
example,
on
the
java
side,
you
can
have
a
java
exporter
that
sends
directly
to
your
back-end
monitoring
system
or,
if
you're,
going
through
the
collector
that
steve
talked
about
the
back-end
monitoring
systems,
they
only
have
to
write
their
exporter
once
for
the
collector
and
then
the
java,
auto
instrumentation
flows
to
that
back-end
system.
Same
with
other
language,
instrumentations.
B
And
speaking
of
other
languages,
so
auto
instrumentation
is
growing.
We've
got
a
javascript
community,
python,
community,
ruby,
community
and
dotnet
as
just
getting
going.
So
this
is
a
the
instrumentation
is
a
great
way
to
get
involved
in
the
community,
because
you
know
you
can
take
your
favorite
language,
your
favorite
library,
and
go
and
see
how
you
can
contribute
instrumentation
for
that
particular
library.
A
So
to
kick
us
off
here,
I'm
actually
going
to
start
up
a
docker
compose
file,
and
while
it
is
booting
up
I'll
walk
you
through
exactly
what
it's
doing.
So
what
I've
done
is
I've
taken
the
spring
pet
clinic
micro
services
repository
and
it
actually
contains
a
docker
compose
file
made
up
of
multiple
java.
Microservices
is
kind
of
outlined
in
this
architecture.
Diagram
that
you
see
here.
A
A
So
from
a
diff
perspective.
Basically,
what
we
did
is
add
a
collector.yaml
file,
so
you
actually
configure
the
opencylindry
collector
through
yaml,
and
here
you'll
see
that
we
have
configured
a
receiver,
basically
the
otop1,
which
is
the
default
one.
That
java,
auto
instrumentation
will
send
data
out,
as
so
we're
just
receiving
over
otop,
and
then
we
are
going
to
export
to
a
zipkin
endpoint.
That
is
a
different
docker
file
that
is
running
inside
of
this
microservices
based
architecture.
A
So
that's
just
a
one-time
config
for
the
collector
again,
if
I
want
to
add
different
processors,
I
can
go
ahead
and
do
that
here.
In
this
case,
it's
just
a
pretty
standard
configuration
next
up.
We
went
through
the
docker
compose
file
and,
as
trust
mentioned,
you
basically
just
add
this
java
agent
flag.
So
you
have
to
go
ahead
and
tell
it
where
the
jar
is
for
the
open
symmetry
java
agent
and
then
specify
what
the
service
name
is.
So
we
also
set
an
environmental
variable
so
for
each
of
the
docker
containers.
A
So
that's
really
the
only
changes
here,
you'll
notice,
no
java
changes,
no
recompiling
of
the
code.
We've
just
pulled
down
this
jar
specified
the
java
agent
set
the
service
through
an
environmental
variable
and
deployed
the
open
symmetry
collector,
and
the
net
result
should
be
that
we
see
telemetry
data
coming
through
our
system.
A
So
let's
go
ahead
and
see
if
that's
happening.
So
we
are
in
a
zipkin
here
we'll
go
ahead
and
look
at
the
service
name,
and
you
can
see
that
services
are
starting
to
appear
already.
We
can
go
ahead
and
look
for
individual
traces
and
you
can
see
the
traces
are
starting
to
come
into
the
system.
So
here
we
have
a
discovery,
server
or
service.
That's
calling
a
config
service.
You
can
see
that
this
is
actually
using
the
open,
telemetry
sdk
and
we
actually
get
some
semantic
conventions
and
metadata
automatically
provided
for
us.
A
So
you
can
see
this
is
restful
services.
We
can
see
the
status
code
we
can
see
which
methods
being
used
so
again,
some
pretty
rich
information.
If
there
were
errors
like
400,
500
level
errors,
they
would
also
show
up
here
and
as
the
app
gets
hit
more
you'll
actually
see
that
a
variety
of
additional
micro
services
will
start
to
show
up.
As
you
can
see
now
we
actually
have
several
calls
coming
into
the
system,
and
so
each
individual
request,
no
matter
which
microservices
it
hits
it'll
go
ahead
and
generate
trace
data.
A
For
that
so
again,
very
simple
to
get
started.
You
can
take
basically
any
java
application.
That's
using
one
of
the
supported
libraries
that
trash
showed
earlier,
go
ahead
and
add
the
the
jar
and
collector
and
you
can
go
ahead
and
send
this
to
your
back
end
or
you
can
even
log
it
locally
and
just
see
what
data
is
being
generated.
It's
pretty
easy
to
extend
this
like.
B
Easy
yeah,
so
we're
steve
and
I
were
both
super
excited
to
share
the
open,
telemetry
work,
that's
going
on
with
the
folks
with
new
folks
and
we
would
love
to
have
you
join
the
join.
The
conversation
join
in
the
project,
whether
that's
as
a
testing
it
and
providing
feedback
providing
code.
There's
a
couple
ways
to
get
started.
There's
a
couple
of
gitter
channels.
If
you
post
there
you'll
find
a
bunch
of
a
bunch
of
people
monitor
those
bunch
of
friendly
folks
join
a
sig.
B
If
you
go
to
this
link,
you
can
see,
find
your
again
find
your
favorite
programming
language
or
your
favorite
topic
like
logging
or
metrics,
and
there's
weekly
sig
meetings
and
there's
and
yeah
and,
like
I
mentioned
earlier,
instrumentation
is
a
a
great
way
to
get
started.
I
think
biased.
You
know
again
take
your
favorite
language.
Take
your
favorite
libraries
go
test
it
out.