►
Description
Don’t miss out! Join us at our upcoming hybrid event: KubeCon + CloudNativeCon North America 2022 from October 24-28 in Detroit (and online!). Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
OpenTelemetry: The Vision, Reality, and How to Get Started - Dotan Horovits, Logz.io
Everyone wants observability into their system, but find themselves with too many vendors and tools, each with its own API, SDK, agents and collectors. In this talk Horovits will present OpenTelemetry, an ambitious open source project with the promise of a unified framework for collecting observability data. With OpenTelemetry you could instrument your application in a vendor-agnostic way, and then analyze the telemetry data in your backend tool of choice, whether Prometheus, Jaeger, Zipkin, or others. Horovits will cover the current state of the various projects comprising OpenTelemetry (across programming languages, exporters, receivers, protocols and more), some of which are not even GA yet, and provide practical guidance on how to get started with OpenTelemetry in your own system.
A
Hello,
everyone
glad
to
be
here
and
amazing
to
see
a
full
house.
There
are
a
few
seats
in
the
at
the
front.
So
if
you
can
get
some
more
people,
there
are
definitely
more
seats
but
amazing
to
see
everyone
here
and
a
full
house
shame.
We
can't
see
the
people
there
on
the
virtual
platform
and
really
excited
to
be
here
at
kubecon,
and
today,
I'd
like
to
talk
to
you
about
open
telemetry,
a
project
that
I'm
very
passionate
about
the
vision,
the
reality
and
how
to
get
started
with
it.
A
A
A
I'm
an
advocate
of
open
source
and
communities
in
general
and
the
cncf
in
particular.
That's
why
it's
very
exciting
to
be
here
on
the
kubecon
stage,
I
co-organized
the
local
cncf
chapter
in
tel
aviv.
I
organize
devops
days.
I
have
a
podcast
open
observability
talks
on
open
source,
devops
and
observability.
So
if
your
podcast
fans
do
check
it
out
and
in
general,
you
can
find
me
everywhere
at
horvitz
and
let's
talk
about
observability.
A
As
you
all
know,
observability
is
essentially
the
ability
to
understand
the
state
of
our
system
based
on
the
telemetry
data
that
it
emits
and
the
vision
talks
about
unified
observability
across
different
signal
types,
classically,
the
three
pillars:
logs
metrics
and
traces,
but
also
across
different
sources.
So
again
think
about
your
front-end
code,
your
back-end
code,
your
kafka
red
these
other
open
source
projects,
cloud
services,
essentially
just
fusing
all
of
these
together
to
gain
insights
and
observability
into
your
system.
A
So
that's
what
open
telemetry
comes
to
solve
in
a
nutshell,
so
open
telemetry
or
hotel
as
its
nickname.
So,
if
I
say
oh
tell
do
forgive
me.
That's
how
usually
the
group
internally
calls.
It
is
an
observability
framework
for
generating
and
capturing
collecting
telemetry
data
from
systems
cloud
native
systems
across
logs
metrics
and
traces,
so
one
one
framework
to
rule
them
all.
A
If
you'd
like
and
it's
a
ua
project
under
the
cncf,
it's
an
incubating
project-
it's
in
essence,
in
fact,
a
merge
of
open
tracing
and
open
census
projects
of
the
cncf.
So
if
you're
familiar
with
them,
that's
the
future
path.
It's
you
need
to
move.
That's
that's
especially
interesting
for
you.
This
talk,
and
I'm
very
happy
to
be
here,
especially
here
on
kubecon
stage,
to
say
that
open
telemetry
has
been
vastly
adopted
across
the
industry.
So
you
see
all
the
cloud
vendors,
all
the
monitoring
and
observability
solutions.
A
So
you
can
go
check
it
out
yourselves
and
slice
and
dice
the
data,
but
essentially
it's
pretty
consistent
for
for
a
lot
of
time
for
a
long
time
already,
and
I
hope
that
I
convinced
you
that
open
telemetry
is
interesting
and
that
you
are
and
the
reason
that
this
is
important,
because
this
is
how
projects
become
de
facto
standards.
When
you
see
the
industry
aligning
behind
it.
A
And
then
that's
the
the
bluish
part
on
the
on
the
left
hand,
side
and
then
a
unified
way
of
collecting
and
processing
that
data
across
different
sources,
both
our
apps
and
other
types
of
sources,
infrastructure
and
others.
And
then
that's
the
green
part
in
the
middle
and
then
a
standard
way
of
exposing
and
transmitting
that
data,
the
telemetry
data,
that's
the
orange
part
and
then
sending
it
off
to
whichever,
whichever
back
end
you
choose
open,
telemetry
does
not
take
any
stand
in
the
back
end
europe
to
it's.
A
A
So
it
defines
the
api
spec,
the
sdk
spec
and
the
data
spec
things
such
as
semantic
conventions,
annotations
and
so
on
across
logs
metrics
and
traces,
and
though
you
won't
probably
as
end
users
interact
directly
with
the
specification
on
a
regular
basis
as
much
as
the
other
components.
It
is
important
to
understand
that,
because
this
comes
to
solve
the
exact
problem
we
talked
about
before,
of
fragmentation
of
each
tool
and
each
vendor
having
its
own
and
each
programming.
A
Language
has
having
its
own
apis,
sdks
formats
and
so
on,
and,
moreover,
having
that
across
one
platform
allows
then
the
correlation
between
the
data.
So
it's
not
just
one,
it's
one
that
can
correlate
across
signals
and
across
sources.
So
that's
very
important
to
to
be
aware
of,
and
the
first
component
that
most
people
encounter
are
the
client
libraries
open.
Telemetry
provides
one
api
and
one
sdk
per
language
with
which
you
can
instrument
and
extract
logs
metrics
traces
from
your
application,
of
course,
adhering
to
the
specification
that
we
talked
about
before.
A
It
also
provides
integrations
with
popular
libraries
and
frameworks,
our
pc
storage
web
frameworks
and
so
on,
for
per
language,
of
course,
so
that
they
can
extract
from
these
languages,
more
information
and
propagate
context
and
so
on.
Within
these
frameworks,
and
also
auto
instrumentation
agents
per
language,
depending
on
the
languages
that
allow
for
a
low
code
to
maybe
no
code
instrumentation,
so
that
you
don't
need
to
modify
your
code
to
start
extracting
at
least
table
stakes,
type
of
metrics
and
and
data.
A
So
essentially
open.
Telemetry's
mission
statement
is
to
allow
the
full
range
between
the
fully
manual
instrumentation
to
the
fully
automated
instrumentation
and
anything
in
between
and
that's
important
to,
say
anything
in
between.
Because
you
can
combine,
you
don't
need
to
decide
either
or
you
can
start
with
auto
instrumentation
to
get
some
table
stakes
and
also
lower
barrier
to
entry.
And
then
you
can
add,
on
top
of
that
custom,
instrumentation
to
add
more
fine-grained
zoom
in
in
areas
that
you
find
important
enough
and
also
it
accommodates
to
different
situations.
A
If
I
have
a
something
that
I
can't
modify
the
code,
I
can
I
need
to
monitor
as
black
box
monitoring,
because
I
can't
modify
or
I'm
not
allowed
to
modify
or
it's
a
third
party.
Then,
for
example,
auto
instrumentation
would
be
the
only
option.
It
would
be
a
must-have,
so
you
get
the
full
range
of
the
instrumentation
capabilities.
A
Next
up
I'd
like
to
talk
about
open,
telemetry
collector
that
can
collect
telemetry
both
from
the
sdks
that
we
talked
about
before,
but
also
from
other
sources.
It
could
collect
from
infrastructure
components,
open
source
cloud
services.
I
don't
know
you
have
a
kafka
running,
you
want
to
send
it.
You
have
my
sequel,
you
want,
I
don't
know
aws
or
azure,
and
google
services,
whatever
you
can
collect
and
the
collector
can
support
that
process.
A
It
aggregate
and
then
send
it
to
whichever
backend
you
choose
and
as
you
could
imagine,
you
can
see
that
here
also
it's
built
as
a
typical
data
pipeline
process
data
processing
pipeline.
So
it
has
a
receivers
in
multiple
protocols,
it
has
processors
and
then
it
has
exporters
in
multiple
protocols.
So,
just
as
an
example,
if
my
code
emits
traces
in
a
jaeger
format,
I
just
plug
in
the
jaeger
receiver.
If
I
want
a
metrics
from
kafka,
I
plug
in
the
kafka,
receiver
and
so
on
and
so
forth.
A
That's
for
the
receivers,
then
the
processors
have
many
types
of
processors
to
do
different
types
of
aggregations
and
processing,
batching,
filtering
sampling
and
so
on,
and
I
can
also
chain
concatenate
different
processes
to
create
more
elaborate
logic
and
then
based
on
the
backend
analytics
tool
that
I
use
and
it
doesn't
have
to
be
a
single
tool.
I
can
use
several
I
plug
in
the
relevant
exporters.
I
can,
if
I
want
to
send
it
to,
I
don't
know
aws
x-ray.
I
I
have
the
receiver
for
that.
I
want
to
send
it
to
a
google
pub
sub.
A
A
And
the
last
component
that
I
would
like
to
talk
about
is
the
open
telemetry
protocol
or
otlp
and
otlp
is
a
essentially
a
general
purpose.
Telemetry
data
delivery
protocol.
It
can
be
used,
as
you
can
see
here,
to
send
between
the
sdk,
the
the
hotel
sdk
to
the
hotel
collector.
It
can
be
used
to
send
between
the
collector
and
the
backend
analytics
tool.
If
it's
it
supports
it,
it
can
be
used
between
intermediary
nodes
such
as
two
different
collectors
or
any
other
purpose.
It's
really
a
general
purpose
protocol.
A
It
currently
supports
the
binary,
protobuf,
encoding
and
they're
in
the
works.
There's
a
plan
to
support
also
json
encoding
over
http,
so
you'll
have
json
over
otlp
over
http
and
it's
agnostic.
So,
as
you
can
see,
you
can
actually
take
the
protofile
and
you
can
generate
your
own
grpc
clients
yourselves.
If
you
so
choose,
that's
part
of
the
the
freedom
that
you
have
when
you
use
this
you're,
not
locked
into
even
using
these
implementations
that
open
telemetry
provides
it's
part
of
the
mindset
of
open
telemetry.
A
Another
important
point
that
I'd
like
to
make
about
the
mindset
is
that
open
telemetry
as
a
project
does
not
mandate
you
to
use
otlp
protocol,
as
I've
shown
you
just
now
with
the
collector.
The
collector
supports
many
protocols
for
the
receivers
for
the
ingest
many
protocols
for
the
exporters
for
the
egress,
so
you're
not
bound
to
it.
A
However,
as
an
as
a
holistic
framework
aiming
to
provide
a
holistic
way
of
generating
and
collecting
telemetry
data,
the
purpose
is
to
get
the
industry
into
one
unified
protocol.
That
can
then
have
also
the
benefit
of
correlating
data,
because
if
we
send
logs
metrics
and
traces
together
with
the
unified
data
model,
then
we
can
also
create
the
correlation
and
we
can
align
the
semantic
conventions
and
so
on.
So
that's
the
mindset
and
that's
the
goal
and
the
same
goes
by
the
way
to
other
components
as
well.
A
You
can
use
the
sdk
auto
sdk
without
the
collector.
For
example,
sending
from
the
sdk
directly
to
a
back-end
analytics
tool
you
can
use
the
collector
without
an
sdk
like.
I
gave
the
example
of
collecting
from
kafka
directly
and
so
on.
So
it's
a
loosely
coupled
yet
a
holistic
framework
aiming
to
provide
a
standard,
unified
way
of
generating
and
collecting
telemetry
data,
that's
about
otlp
and
maybe
the
mindset.
A
These
are
the
main
components
that
I'll
cover
there
are
other
components
and
other
elements,
there's
a
a
open,
telemetry
operator
that
allows
you
to
install
that
easily
on
kubernetes
and
other
components.
But
for
the
sake
of
this
discussion
I
will
keep
it
here
and
now
I
would
like
to
talk
about
so.
Okay.
What's
the
what's
the
state
of
open,
telemetry
project-
and
most
importantly,
is
that
ga
can
I
use
it
in
production?
That's
the
most
interesting
question
right,
so
show
of
hands
who
thinks
yes.
A
A
So
I'm
oversimplifying
even
that,
but
just
to
get
the
drift
and
each
sig
in
each
special
interest
group
and
each
working
group
and
each
such
component
has
its
own
development
life
cycle,
which
means
the
different
parts
of
open
telemetry
may
be
in
different
state
of
the
maturity
life
cycle,
which
is,
in
the
cncf
terms,
draft
experimental,
stable
and
deprecated.
Just
for
those
who
are
more
new,
we
have
like
67
percent
new
people
there
this
year
in
kubecon,
it's
very
exciting
and
just
to
align
the
terms
with
the
common
terms
in
the
industry.
A
A
A
So,
okay,
we
understand
that
it's
complicated.
Thank
you
dutan,
but
still
what's
the
state
of
open
telemetry,
and
for
that
I'd
like
to
break
it
down
by
signal
types
and
the
first
signal
and
the
most
true
one
is
traces
and
open.
Telemetry
is
generally
available
for
traces
since
last
year,
which
means
that
the
tracing
api
and
sdk
and
the
protocol
specifications
are
stable,
the
collector's
table.
We
have
many
client
libraries,
sdk
implementations
that
are
version
1.0
or
above
version.
1.
is
when
the
tracing
implementation
is
complete.
A
So,
as
you
can
see
here,
we
have
for
java
and
go
and
dotnet
and
python
and
c,
plus,
plus
and
javascript,
and
ruby
and
erlang
and
swift
and
working
on
on
more
and
it
keeps
on
advancing
so
rapidly
that
maybe
that
is
even
advanced.
Since,
since
the
last
time
I
made
the
slide-
and
most
importantly,
as
I
said,
ga
means
that
it
comes
with
guarantees
for
a
long-term
support
for
backwards
compatibility
for
dependency,
isolation.
What
you'll
be
expecting
to
run
it
in
production?
A
Just
the
announcement
on
kubecon
and
release
candidate,
again
just
to
align
terms,
means
that
it's
practically
ga
now
collecting
the
feedback
from
users
and
if
nothing
major
critical
comes
up,
it
should
be
turning.
The
same
versions
will
return
ga
within
a
matter
of
a
week
or
a
couple
of
weeks.
More,
that's
that's
the
time
frame,
so
the
api,
the
sdk,
the
protocol
are
obviously
stable.
The
api
and
sdk
specifications
are
already
implemented
in
java
and
dot
net
and
a
python
in
rc
in
release.
Candidate
javascript
is
just
a
week
away
or
something
like
that.
A
So
and
then
many
more
languages
will
join
in
the
coming
month
or
two
in
terms
of
the
collector.
The
collector
supports
metric
pipelines
and
very
important
to
say
because,
as
you
also
heard
in
the
previous
talk
here
in
this
hall
and
generally
prometheus
being
a
de
facto
standard
in
the
in
the
metric
side,
so
there's
prometheus
support
that
has
been
worked
on
in
collaboration
with
the
prometheus
community.
A
I
think
it's
a
great
example
of
collaboration
under
the
cncf,
so
applause
to
the
to
the
teams
involved,
and
it
means
that
the
sdk
has
exporters
in
prometheus
format.
The
collect
the
auto
collector
has
receivers
and
exporters
in
prometheus
formats.
From
ethiopia's
remote
right,
the
otlp
to
prometheus
specification
is
aligned
the
data
model.
So
that's
that's
there
and
the
least,
I
guess
advanced
signal
is
logs
that
are
still
experimental.
We
hope
to
have
that
gad
this
year
and
with
logging.
It's
important
to
to
understand.
A
Login
is
the
most
long-standing
signal.
Obviously,
so
everyone
has
logging.
Every
system
has
logging,
it's
been
there
for
ages,
so
we
can't
just
ignore
that.
Obviously,
so
the
first
focus
is
to
align
with
existing
and
support
existing
logging
sources
and
logging
systems,
and
for
that
there's
also
work
around
around
there.
A
Sorry
around
the
pendants
logopenders
that
are
under
development
in
in
many
languages
so
that
even
existing
logs
that
are
typically
text
based,
unstructured,
maybe
even
file
based,
can
still
be
augmented
with
additional
data
such
as
a
trace
id
and
other
important
data
to
allow
correlation,
obviously,
obviously
sending
that
over
otlp,
so
even
for
existing
logging
sources
being
able
to
ingest
and
then
model
it
over
all
tlp
and
send
it
alongside
metrics
and
traces.
That's
the
the
first
priority,
but
then
following
up
as
a
holistic
framework.
A
As
we
talked
about
the
mission
statement,
we
were
working
to
build
a
new
strongly
typed
and
machine
readable
format
also
for
logs.
So
it's
not
discarded.
It's
definitely
the
end
goal
and
to
try
and
converge
the
industry
also
around
that,
and
I
think
the
most.
If
you
look
at
the
components
that,
for
example,
you
see
that
the
protocol
is
is
stable
and
can
send
that's
actually
also
relatively
new
news.
A
I
think
less
than
a
week
ago,
we've
announced
that
the
pro
the
otlp
protocol
for
logs
is
already
stable
and
supports
the
data
model
that
has
been
agreed.
I
think
a
month
or
two
ago
and
the
and
the
sdk
is
experimental
and
can
already
transmit
over
otlp
one.
One
thing
that
is
important
is
that
the
api,
for
example,
is
left
to
last.
It's
still
draft,
because
again
we
haven't
prioritized
inventing
new
apis.
For
that.
A
A
What
should
the
structured,
well
structured,
strongly
typed
log
look
like
and
for
that
also
exciting
news
from
the
recent
months
is
the
collaboration
with
ecs
elastic
common
schema
for
those
who
know
that
so
collaboration
between
the
communities,
essentially
the
open
source
communities
to
get
all
the
aggregated
knowledge
and
work-
that's
been
done
around
the
elastic
common
schema
and
merge
it
together
with
open,
telemetry
and
and
join
forces
around
that.
So
really
exciting
news
about
that
as
well.
A
A
A
lot
of
hard
work:
we
have
some
maintainers
here
at
the
audience,
so
we
all
owe
them
a
great
deal
of
gratitude
for
for
achieving
this
very
important
milestone.
Also
for
those
who
worked
on
the
logging,
as
I
said,
for
getting
the
data
and
protocol
stable
and
for
those
who
are
interested
in
more
about
the
roadmap
and
the
future
path.
It's
a
lot
less.
A
A
For
example,
we
use
java
with
with
a
spring,
and
then
we
use
the
front-end
node.js
with
happy
and
express
so
it's
important,
because
that
will
help
you
determine
which
sdks
and
agents
you
can
use
and
need
to
use
for
your
your
application,
then,
which
signals
you
intend
to
collect
from
your
system
and
logs
metrics
traces
and
also
in
which
formats,
which
is
particularly
important.
If
it's
a
brownfield
project,
you
already
have
components
out
there
emitting
a
telemetry.
A
Under
the
cncf,
the
open,
telemetry
dot
io
status,
that's
a
very
good
starting
point
to
see
the
high
level
status
overview.
There
are
many
other
useful
resources
on
that
site,
like
the
the
docs,
obviously
and
there's
a
also
relatively
new
registry
that
you
can
actually
search
for
your
stack.
So
you
realize
that
you
need,
I
don't
know.net,
you,
google,
you
search.net,
and
then
it
will
take
you
to
the
specific
links
on
the
github
repo
that
that
pertain
to
that.
So
it's
a
really
helpful
way
to
navigating
around
the
massive
github
ripple
there.
A
A
So
I
hope
that
you
find
it
useful
and
by
the
way,
if
you,
if
you
do,
have
feedback
on
the
on
the
guide
or
if
you
have
feedback
on
this
talk
or
if
you
have
any
any
questions,
we're
now
ever
going
to
have
a
q
a
so
do
don't
go.
We
have
time
for
you
to
ask
and
also
for
the
people
from
the
from
the
virtual
platform
and
then
feel
free
to
reach
out
to
me
also
after
at
horvitz,
and
with
that.
Thank
you
very
much
and
don't
go.
A
A
A
It's
based
on
on
others
called
snapshots,
but
ones
that
are
based
on
that
are
modeled
can
be
modeled
after
as
events
similar
to
logs,
it
can
be
converted
and
then
relayed
over
the
same
mechanism
that
is
built
for
for
logs.
So
it's
interesting,
maybe
to
explore
adapters
that
can
actually
transform
that,
but
I
don't
know
of
any
specific
goal
to
adhere
specifically
to
these
these
formats.
Again,
if
we
have
one
of
the
maintainers
wanting
have
any
more
detailed
information
feel
free
to
jump
in.
I
hope
I
answered
the
question.
Yeah
thanks.
A
A
So
most
of
what
you
see
and
what
you've
seen
is
really
back
back-end-based.
Of
course
you
can
take
the
existing
javascript,
sdks
and
apis,
and
you
can
work
it
also
with
client-side,
but
client-side
does
have
its
own
constraints.
You
need
the
session
id
propagated
and
things
like
that
that
haven't
been
modeled
into
the
specification.
A
So
there
is
a
new
working
group
for
the
client
instrumentation
that
aims
strictly
squarely
at
these
use
cases
of
instrumenting
web
pages,
web
apps,
mobile,
apps
and
so
on,
and
so
I
think
this
would
fit
perfectly
into
both
synthetic
synthetic
monitoring
and
and
real
real
user
monitoring
things
that
look
at
your
system
from
outside
as
a
black
box.
So
if
you
are
interested
in
that
do
get
involved
check
out
this
new
working
group,
it's
really
in
the
early
phases,
so
you
can
actually
get
involved
and
and
and
influence.
A
Prometheus
exported
from
the
application,
so,
as
I
said,
the
collector,
the
open,
telemetry
collector,
he
supports
the
prometheus
format,
so
you
can
actually
collect
it
with
the
hotel
collector
and
you
can
relay
that
over
otlp.
So
there
is,
there
is
a
collaboration
that
has
been
done
to
support
prometheus
in
on
the
collection
side.
As
I
said,
you
can
also
use
prometheus
as
a
back-end,
so
you
can
also
use
a
remote
right
exporter
to
send
it
to
prometheus
as
a
back-end,
so
you're
supported
both
both
ways.