►
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2023 in Amsterdam, The Netherlands from April 17-21. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
OpenTelemetry or eBPF? That is the Question - Omid Azizi, New Relic (Pixie)
In the observability space, OpenTelemetry and eBPF are two technologies that have been rapidly changing the landscape. So which should you use? The OpenTelemetry project provides a rich set of tools with which teams can instrument their applications, enabling deep visibility into the application behavior. eBPF, on the other hand, has been powering instrumentation-less observability through other projects like the CNCF Pixie and Hubble projects. In this environment, users often wonder which approach they should turn to. In this session, we'll cover the strengths and weaknesses of both approaches, and show how both approaches have a role to play. We'll demonstrate how eBPF observability tools can be configured to export to OpenTelemetry collectors as automatic data sources. We'll then focus on the problem of tracing, and how request tracing works with eBPF and OpenTelemetry. In this process, we'll show how eBPF has the power to avoid some manual instrumentation; in contrast, we'll show how instrumentation is still required for true distributed tracing today. The session will wrap up with a perspective into the future of the two technologies, and what is on the horizon.
A
So
today
my
talk,
it's
gonna
be
on
open,
Telemetry
or
ebpf.
That
is
the
question
right,
so
I'll
get
to
that
in
a
second
and
we'll
get
to
that
question
in
a
second.
But
for
someone
to
introduce
myself,
I'm
Omid
Azizi
I
work
on
pixie
I'm,
a
principal
engineer
at
New
Relic
was
a
founding
engineer
at
pixie.
A
Leading
a
lot
of
the
ebpf
work
happening
there,
and
so
we
do
a
lot
of
observability
work,
and
so
why
this
talk
like
why?
Why
did
I
decide
to
talk
about
this?
So
I
was
actually
at
kubecon
EU
in
Valencia,
and
this
question
just
came
up
a
lot
whether
it
was
in
in
the
conference
sessions
or
at
the
booths.
A
It's
it's
like
well,
there's
a
lot
of
confusions
like
what
should
I
be
using
for
my
observability
Solutions
should
I
be
using
ebpf
based
Solutions
should
I
be
using
open,
Telemetry,
there's
a
lot
of
Buzz
around
both
what
am
I
supposed
to
do.
Right
and
I
realize
it's
really
like
confusing
to
the
outside.
What
which
one
of
these
Frameworks
we
should
be
using,
and
both
projects
are
really
so
massive
that
it's
it's
really
hard
to
navigate
right.
A
So
point
of
this
talk
is
to
kind
of
dive
in
a
little
bit
and
see
so
before
we
kind
of
Go
In,
Too,
Deep
I
just
want
to
level
Set
so
that
we're
kind
of
all
working
off
the
same
Baseline,
Thomas,
already
kind
of
did
the
introduction
to
evpf
so
I'm,
just
going
to
briefly
mention
that
the
landscape
is
huge
with
evpf
right.
Ebpf
is
so
many
different
things
to
different
people.
I
remember
at
the
last
cubecon
in
Los
Angeles.
A
You
know
there
was
this
analogy
last
cb50
in
Los
Angeles.
There
was
this
analogy
that
you
know.
Evpf
is
like
an
elephant
where
it's
like
the
ear,
the
the
trunk,
the
tail,
they
all
look
kind
of
different,
and
it
does
so
many
different
things.
Yeah
it
does
like
networking
stuff.
We
can
do
observability
with
it.
We
can
do
security
now.
For
the
purposes
of
this
talk,
obviously
we're
talking
about
open
Telemetry,
that's
in
the
observability
space,
so
we're
going
to
be
focusing
mostly
on
ebpf
versus
open
Telemetry,
the
observability
domain.
A
Just
like
ebpf
has
a
lot
of
things.
Open
Telemetry
is
a
lot
of
things,
and
so
we
can
break
it
down
into
like
three
main
components
of
what
open
Telemetry
provides.
The
first
thing
is,
it
provides
a
standard.
It
provides
standardized
data
formats
where
you
can
transfer
data,
Telemetry
data,
observability
data
and
exchange
between
different
different
layers.
A
The
second
thing
is
that
it
provides
a
toolkit,
so
there's
an
SDK,
that's
language
specific.
So
if
you're
working
in
Java
there's
a
Java
SDK,
if
you're
working
in
Python
there's
a
python
SDK
with
which
you
can
go
and
instrument
your
own
applications
and
then
the
third
major
component
is
kind
of
more
infrastructure
that
they
provide
for
doing
like
collecting
all
the
the
Telemetry
data
processing
it
manipulating
it
and
then
exporting
it
out.
A
I
think
it
makes
sense
to
kind
of
look
at
a
diagram
for
this
to
kind
of
get
better
grasp
of.
What's
actually
going
up
going
on
a
typical,
open,
Telemetry
setup.
You'll
have
your
cluster
you'll?
Have
your
services,
your
application
running
there
you
will
have.
You
will
have
some
open,
Telemetry
instrumentation
on
the
various
different
services
that
you
have
running
right.
A
Those
will
be
sending
up
Telemetry
data
through
various
supported
protocols,
but
the
otlp
is
kind
of
the
Native
open
Telemetry
protocol
and
those
go
to
The
Collector
that
we
just
mentioned.
The
Collector
is
aggregating
all
the
data
and
then
exporting
it
out
to
a
back
end,
I,
actually
think
of
the
back
end
more
of
a
really
a
front
end,
because
that's
the
part
where
you,
as
the
user
of
the
Telemetry
data,
are
going
to
be
working
with
the
data.
A
Okay,
so
that's
just
for
level
setting
observably
all
starts
with
instrumentation.
So
let's
talk
about
instrumentation
when
we're
talking
about
instrumentation,
there's,
it's
useful
to
kind
of
look
at
the
the
software
stack.
We
have
the
operating
system
at
the
bottom,
which
is
the
broadest
layer.
On
top
of
that,
we
have
various
different
libraries
and
then,
on
top
of
that,
we
have
our
applications
that
we
write
when
you're
trying
to
choose
an
observability
approach,
there's
kind
of
these
two
opposing
forces.
A
On
the
other
hand,
the
lower
that
you
go,
you
lose
context
right
and
so
there's
this
competing
Force
saying
that
you
should
instrument
things
as
high
up
as
you
can
in
the
softer
stack
so
that
you
have
as
much
context
as
you
can.
So
you
know
what's
going
on
right,
so
a
big
theme
of
this
talk
is
kind
of
going
to
be.
A
That
is
really
this
just
this
contention
between
going
as
low
as
you
can
in
the
SEC,
so
go
as
low
as
you
can
in
the
stack,
because
that's
where
you're
going
to
have
the
broadest
visibility,
but
not
so
low
that
you
lose
the
information
that
you
need
in
terms
of
open,
Telemetry
and
abpf.
Both
are
effective
tools
for
instrumenting
and
getting
Telemetry
data
out.
Ebpf
has
the
advantage
that
it
can
also
hook
into
the
operating
system.
A
A
A
What
automatic
instrumentation
means
is
it's
not
that
somebody
that
that
the
instrumentation
happened
automatically
somebody
had
to
do
the
instrumentation,
it's
just
that
you
didn't
have
to
do
the
instrumentation
right,
so
somebody
else
went
through
the
hard
effort
of
doing
that
instrumentation
for
you
and
you
reap
the
benefits
right.
So
if
somebody
goes
in
and
does
instrumentation
in
with
ebpf-
and
you
just
use
that
great
right,
you
get
a
bunch
of
visibility
and
you
didn't
have
to
do
a
whole
bunch
of
hard
work
same
thing
with
instrumentation
that
may
occur
in
libraries.
A
If
you
have
to
go
and
do
sometimes,
you
do
need
to
like
kind
of
pull
out
your
tools
and
do
some
more
manual
work
and
that's
manual.
Instrumentation
that'll
typically
be
in
the
application
layer.
We'll
come
back
to
manual
instrumentation
a
little
bit
later
on.
In
the
talk
for
this
part
kind
of
for
the
broad
General
audience,
you
should
be
looking
for
solutions
that
can
give
you
automatic
instrumentation.
If
you
can,
if
you're
just
looking
for
Solutions,
you
should
be
seeing
what
works
out
of
the
box
rather
than
Reinventing
anything.
A
Just
some
quick
examples
of
what
automatic
instrumentation
looks
like.
If
you
look
at
open
Telemetry,
you
can
run
Java
with
a
pre-existing
open,
Telemetry
Java
agent.
You
don't
have
to
change
anything
about
your
application.
You
just
deploy
it
with
that
agent
enabled
and
you'll
automatically
get
observability
into
a
bunch
of
different
Java
Frameworks
like
if
you're
using
spring
in
Java
spring
boot
in
Java,
or
maybe
you
have
Kafka
or
whatever
it
just
they've
they've
done
the
hard
work
for
you.
A
You'll
get
that
instrumentation
out
of
the
box
same
thing
with
evpf
in
terms
of
automatic
instrumentation
right.
You
have
your
application
running.
You
can
use
things,
there's
a
bunch,
but
you
can
use
Hubble
from
from
psyllium.
You
can
use
Pixi
and
you'll
get
a
bunch
of
information
about
what's
actually
happening
in
terms
of
messages
that
are
being
exchanged
within
your
within
your
application
right
so
automatic
you
don't
have
to
do
any
work.
Great.
A
So
what
should
you
be
using
we're
going
to
cover
metrics
a
use
case
from
message
tracing
and
a
use
case
for
profiling
in
terms
of
metrics
first,
so
we
kind
of
already
touched
on
this,
but
there's
this
there's
a
spectrum.
You
can
look
for
observability
metrics
that
are
coming
out
from
the
operating
system,
all
the
way
up
to
the
application
layer
at
the
end
other
end
of
the
spectrum,
if
you're
looking
for
kind
of
os
level
stats
things
like
CPU
utilization
memory,
anything
like
that.
A
A
A
And
so
I
think
both
open,
Telemetry
and
ebpf
provide
great
Solutions
here
generally,
if
you're
looking
at
ebpf
you're
going
to
get
broad
basic
coverage
right
and
it
that's
great.
The
reason
it's
great
is
that
it's
you
don't
need
to
count
on
anyone
having
instrumented
or
deploying
the
application
in
any
special
way.
Ebpf
can
come
in
from
the
outside
and
monitor
for
you
and
it's
kind
of
serves
as
a
safety.
Network
gives
you
that
broad
coverage.
A
Okay,
so
metrics
is
kind
of
straightforward
message:
tracing.
Let's
talk
about
that
for
a
little
bit,
so
the
first
thing
I
have
to
say
about
message:
tracing
is
I
have
to
make
a
point
about
terminology.
So
when
we
talk
about
Trace,
it's
actually
to
me,
it's
a
very
confusing
term,
because
in
the
ebpf
world,
R
Community
right
when
we
say
Trace,
we
typically
mean
we're
tracing
an
event
like
BPF
Trace,
right
we're
tracing
something
that's
happening
in
the
system
right
and
we
have
these
terminologies
of
things
like
Trace
points.
A
Pixie,
for
example,
will
Trace
messages
going
from
one
microservice
to
another,
which
just
means
we're
snooping
that
data
to
the
open,
Telemetry
Community.
If
you
mention
the
word
Trace,
they
typically
mean
distributed
Trace,
which
is
a
slightly
different
concept.
A
distributed.
Trace
is
really
following
a
request,
as
it
goes
from
Hop
to
hop
across
the
different
microservices
within
an
application.
So
with
a
distributed
Trace,
you
typically
have
the
concept
of
spans,
like
you,
click
on
something
in
your
browser
right.
That
starts
off
the
top
level
request.
A
That's
your
user
request
behind
the
scenes.
Maybe
that
needs
to
go
to
a
catalog
service.
That's
going
to
be
a
different
span.
The
catalog
service
might
be
making
a
request
to
a
database.
It's
hopping
over
to
the
database.
That's
another
span
and
it's
just
collecting
all
this
information,
as
the
request
is
making
its
way
through
the
system
collecting
all
that
and
aggregating
it
all
together
and
connecting
it
all
together
and
that's
what
a
distributed.
A
Trace
means
right,
and
so
this
is
important,
because
the
state
of
the
art
today
is
that
distributed
tracing
is
hard
to
do
with
ebpf.
So
why
is
distributed
tracing
hard
so
to
do
distributed
tracing
properly?
You
need
context
propagation
right
we're
about
to
follow
the
request,
as
it
goes
from
Hop
to
hop
across
the
different
microservices.
A
Well,
we
need
to
know
where
that
request
came
from
and
so
typically
there's
a
trace
ID
in
a
request
that
we
need
to
be
able
to
follow,
say
the
client
sends
a
request,
a
get
request
to
a
service
right,
there's
a
trace
ID
in
there.
Somebody
had
to
add
that
Trace
ID
into
the
HTTP
request
right
and
then
it
gets
into
the
service.
A
Maybe
maybe
it's
like
node.js
service,
maybe
there's
a
lot
of
different
threads
and
events
going
on,
so
maybe
it
gets
put
into
a
buffer
for
some
time
then
gets
picked
up
later
and
it
gets
sent
a
request
to
a
database
again.
We
need
to
have
the
trace
ID
following
that
request,
all
the
way
through
throughout
the
code,
to
be
able
to
make
this
Association,
and
so,
if
you
take
the
approach
of
I
want
to
use
ebpf
for
observability,
but
I
don't
want
to
modify
the
code
in
any
way.
That
makes
this
context
propagation
difficult.
A
So
to
kind
of
summarize,
the
state
of
message,
tracing
I
would
say
if
you're
looking
at
you
want
to
get
service
maps
and
who's
talking
to
who
ebpf
is
great
at
that.
If
you're
looking
for
metrics
things
like
the
throughput,
the
latency,
the
error
rates,
things
like
that
within
within
that
service
map,
again
ebpf
has
great
at
that.
A
So
is
open
Telemetry
for
both
of
these,
but
if
you're
looking
at
distributed
tracing,
that's
where
open
Telemetry
today,
because
that
context
propagation
can
do
a
little
bit
better
as
a
community
I
think
this
actually
ebpf
Community
I
think
that's
a
challenge
for
us
is:
can
we
can
we
figure
out
how
to
automate
the
distributed
tracing
and
I?
Think
there
are
some
ideas
here,
but
can
we
automate
that
tracing
without
having
to
modify
the
code
in
any
way
to
do
context,
propagation.
A
Okay,
next
use
case
is
profiling
and
I'll
just
briefly
touch
on
this,
but
profiling
is
really
taking
off
in
the
ebpf
community.
We
see
projects
like
pyroscope
parka
Pixi.
They
all
have
profilers
now
ebpf
based
they're
great,
because
they're
low
overhead
they're,
always
on
which
means
a
continuous
profiler
and
and
they're
system-wide.
So
your
profiles,
everything
that's
happening
on
your
on
your
CPU.
A
If
you
had
asked
me
a
year
ago,
you
know,
what's
the?
What
should
you
be
using
open,
Telemetry
or
ebpf
for
profiling?
I'd,
say:
ebpf,
open
Telemetry
doesn't
do
anything
with
profiling,
but
that's
actually
starting
to
change.
There's,
actually
a
work
group
in
the
open,
Telemetry
Community.
That's
trying
to
define
the
standard
so
they're,
not
doing
the
data
Gathering
or
the
profiling
itself,
but
they're,
defining
the
data
standard,
and
so
there's
not
really
going
to
be
much
of
a
competition
here
and
that
data
format
is
not
just
for
ebpf
based
profilers.
A
A
A
few
other
points
I
wanted
to
make
just
some
other
scenarios.
If
you're
looking
for
real-time
debugging
like
something's
wrong,
and
you
don't
have
the
instrumentation
that
you
need,
and
you
really
can't
afford
to
or
don't
want
to
redeploy
your
application
instrument
and
redeploy
your
application.
Okay,
ebpf
wins.
You
know
easily
over
open
Telemetry
there
right,
because
open
Telemetry
is
going
to
require
code
changes.
A
On
the
other
hand,
if
you're
kind
of
dealing
with
interpreted
languages,
it's
it's,
you
can
go
in
with
evpf
and
collect
data
out,
but
you're
going
to
have
to
deal
with
that
layer
of
The
Interpreter,
which
can
be
hard
to
kind
of
figure
out,
and
so
you
know
typically
most
developers,
especially
with
the
ones
that
deal
with
interpreted
languages,
are
going
to
turn
to
open
Telemetry.
When
they're
trying
to
instrument
with
interpreted
languages.
A
A
So
what
if
you
actually
have
to
get
your
hands
dirty
dive
in
and
do
that
instrumentation
yourself?
So
I
wanted
to
start
with
ebpf
and
how
we
can
instrument
with
ebpf.
So
I
have
this
2x2
grid
on
the
y-axis.
There's
the
where
you're
trying
to
do
the
instrumentation.
So
are
you
trying
to
do
in
the
kernel
or
the
user
space
and
on
the
x-axis
we
have
the
instrumentation
type.
A
A
Telemetry
doesn't
really
cover
that
where
this
overlap
is
in
that
bottom
right
corner,
when
you're
trying
to
do
static,
instrumentation,
you're,
trying
to
add
instrumentation
to
code
and
it's
in
user
space
code
and
the
overlap
here,
that's
interesting
is
we
in
the
evpf
community
have
usdts,
and
so
let's
take
a
look
at
that
in
a
little
bit
more
detail.
So
if
you're
trying
to
do
something
with
open,
Telemetry
and
you're
trying
to
manually
instrument,
it's
pretty
straightforward,
like
let's
say
we
have,
this,
handle
request,
function
we're
trying
to
instrument.
A
You
know
you
add
some
code
to
the
beginning.
When
that
request
starts,
you
add
some.
You
know
one
line
of
code
to
when
that
request
ends
and
you're.
You
know
using
open,
Telemetry
you're
going
to
start
generating
these
spans
and
these
events
that
you
can
collect
to
visualize,
what's
happening
in
your
system
right
so
pretty
straightforward.
It
does
nice
context
propagation
for
you
as
well.
That
second
line
there,
where
it
says
with
active
span,
means
any
other
spans
you
create,
after
that,
the
context
will
automatically
get
propagated.
A
If
you
squint
kind
of
right,
you
kind
of
realize
this
kind
of
they
kind
of
look
similar
right.
You
hear
this
example
is
with
Folly.
Like
you
put
a
line
of
code
at
the
beginning
of
your
function,
you
put
a
line
of
code
at
the
bottom
of
your
function
at
the
end
you're
trying
to
in
this
example,
kind
of
look
at
the
span
of
when
that
request
started
when
it
ended
right.
That's
interesting
information
to
you,
perhaps
right!
A
So,
if
you're
doing
manual
instrumentation
with
usdts,
it
looks
a
little
bit
if
you
squint
right,
it
looks
very
similar
to
how
it's
done
with
open
telemetry,
and
so
there
are
some
very
real
differences
between
them
right.
So
you
know
ebpf,
you
have
very
easy
access
to
the
different
members.
If
you
know
how
to
navigate
the
structs,
you
know,
on
the
other
hand,
open
Telemetry
has
that
advantage
of
context
propagation
right.
A
Those
are
pretty
significant
differences,
but
another
question
for
us
as
a
community
is
if,
if
you're
going
to
go
through
that
effort
of
going
to
instrument
your
code,
can
we
somehow
get
the
best
of
both
worlds?
Can
we
get?
You
know
if
have
not
have
a
competition
between
open,
Telemetry
and
ustts
like?
Maybe,
if
you
go
in
an
instrument
with
open
Telemetry,
you
automatically
get
that
usdt
for
free.
A
A
Again,
we
said
anything
in
the
kernel
use
the
BPF.
If
you're
in
the
application
layer
depends
on
your
use
case,
you
can
use
open,
Telemetry
or
evpf.
The
more
specific
you
want
go
open,
Telemetry,
the
more
broad
you
want
to
go,
go
ebpf
message
tracing
again.
If
you're
looking
for
service
maps
and
things
like
that,
you
can
use
both
distributed
tracing.
A
We
mentioned
that's
kind
of
more
the
domain
of
open
Telemetry
today,
profiling,
if
you're
talking
about
the
data
source
of
it
is
going
to
you
know,
evpf
is
a
great
way
to
go
in
terms
of
the
data
format
or
the
standard
with
which
we're
going
to
exchange
Telemetry
information.
Let's
leave
that
to
open
Telemetry
they're
great
at
defining
those
formats.
There's
no
reason
to
reinvent
that
right.
A
So
when
we
kind
of
consider
that
consider
all
that
information,
where
the
strengths
and
weaknesses
of
each
approach
is,
this
is
what
I
would
actually
do
and
I
think
where,
where
things
are
going
to
be
headed
right,
it's
not
so
much
that
open,
Telemetry
or
ebpf,
like
you
ask,
which
should
I
be
using.
It's
not
so
much
that
they're
in
direct
competition
they're
actually
better
together
right.
A
So
if
you
had
that
I
had
this
this
diagram
at
the
the
beginning
of
the
presentation
where
they're
you
know
open,
Telemetry,
setup
of
monitoring
right
and
so,
where
I
think
things
will
be
going,
is
that
you're
going
to
have
your
cluster
you're?
Definitely
still
going
to
have
your
open,
Telemetry
instrumentation?
A
A
Just
a
quick
point
about
the
reality:
I
mean
I
kind
of
touched
on
this,
but
the
reality
software
systems
is
that
they're
complicated,
there's
many
different
microservices
they're
written
in
different
languages
by
different
people.
They
come
from
different
Origins.
You
know
mergers
Acquisitions
things
like
that
projects,
there's
like
come
from
very
different
places,
many
different
Frameworks
being
used.
A
Everyone
wants
to
do
things
for
their
own
way,
because
there's
pros
and
cons
right
and
just
another
Point
here
is
like
it's
really
challenging
to
make
sure
that
in
your
organization
that
everyone's
following
the
instrumentation
rules
of
saying
you
have
to
instrument
things
in
a
certain
way
and
with
certain
Frameworks
like
you
know,
you
have
to
have
the
open,
Telemetry,
Java
agent
running
so
on
and
so
forth.
So
again,
this
is
where
I
think
ebpf
as
a
broad
basic
solution
as
a
safety
net
is
really
important
in
any
observability
holistic.
Observability
solution.
A
All
right
so
last,
two
slides
so
key
takeaways,
really
evpf
open,
Telemetry
they're,
not
they're,
not
really
competing,
there's
some
places
where
there
are
significant
overlap,
but
overall
there's
a
there's.
A
role
for
both
I,
primarily
think
of
ebpf
as
a
data
source
open
Telemetry
provides
that
instrumentation
toolkit.
When
you
need
a
little
bit
more
detail
and
provides
the
data
standard,
you
should
be
using
ebpf
when
you
need
that
broad
coverage,
when
you
can't
rely
on
the
various
different
application
developers
to
have
instrumented
things
properly.
A
It's
especially
important
for
things
like
security,
where
it's
like
you
can't
count
like.
Are
you
going
to
bank
your
security
Solution
on
somebody
having
instrumented
things
properly?
You
can't
really
count
on
that.
Right.
Use,
open
Telemetry
when
that
context,
or
that
application
specific
detail
is
important
and
then
just
to
wrap
up
just
wanted
to
leave
folks
with
food
for
thought.
A
The
first
is
again:
I
mentioned
those
usdts
and
open
Telemetry
instrumentation.
If
we're
going
to
go
in
and
put
in
the
effort
to
instrument
applications,
can
we
get
both
for
this?
You
know
two
for
the
price
of
one
like.
Let
is
there
any
way
that
we
can
unify
these
things.
Can
we
start
thinking
about
it
as
a
community
about
seeing
how
we
can
do
that
to
make
it
better
for
everyone,
and
the
second
is
specifically
to
the
ebpf
community?